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Abstract 

Abstract: This review provides a rough sketch of the evidence, gaps and promising 
directions in e-learning from 2000 onwards, with a particular focus on Canada. We 
searched a wide range of sources and document types to ensure that we 
represented, comprehensively, the arguments surrounding e-learning. Overall, there 
were 2,042 entries in our database, of which we reviewed 1,146, including all the 
Canadian primary research and all scholarly reviews of the literature. In total, there 
were 726 documents included in our review: 235 - general public opinion; 131 - 
trade/practitioners' opinion; 88 - policy documents; 120 - reviews; and 152 - 
primary empirical research. The Argument Catalogue codebook included the following 
eleven classes of variables: 1) Document Source; 2) Areas/Themes of e-learning; 3) 
Value/Impact; 4) Type of evidence; 5) Research design; 6) Area of applicability; 7) 




Pedagogical implementation factors; 8) A-priori attitudes; 9) Types of learners; 10) 
Context; and 11) Technology Factors. 

We examined the data from a number of perspectives, including their quality as 
evidence. In the primary research literature, we examined the kinds of research 
designs that were used. We found that over half of the studies conducted in Canada 
are qualitative in nature, while the rest are split in half between surveys and 
quantitative studies (correlational and experimental). When we looked at the nature 
of the research designs, we found that 51% are qualitative case studies and 15.8% 
are experimental or quasi-experimental studies. It seems that studies that can help 
us understand "what works" in e-learning settings are underrepresented in the 
Canadian research literature. 

The documents were coded to provide data on outcomes of e-learning (we also refer 
to them as "impacts" of e-learning). Outcomes/impacts are the perceived or 
measured benefits of e-learning, whereas predictors are the conditions or features of 
e-learning that can potentially affect the outcomes/impacts. The impacts were coded 
on a positive to negative scale and included: 1) achievement; 2) 
motivation/satisfaction; 3) interactivity/ communication; 4) meeting social 
demands; 5) retention/attrition; 6) learning flexibility; and 7) cost. Based on an 
analysis of the correlations among these impacts, we subsequently collapsed them 
(all but cost) into a single impact scale ranging from -1 to +1. We found, generally, 
that the perception of impact or actual measured impact varies across the types of 
documents. They appear to be lower in general opinion documents, practitioner 
documents and policy making reports than in scholarly reviews and primary research. 
While this represents an expression of hope for positive impact, on the one hand, it 
possibly represents reality, on the other. Where there were sufficient documents to 
examine and code, impact was high across each of the CCL Theme Areas. Health and 
Learning was the highest, with a mean of 0.80 and Elementary/Secondary was the 
lowest, with a mean of 0.77. However, there was no significant difference between 
these means. 

The impact of e-learning and technology use was highest in distance education, 
where its presence is required (Mean = 0.80) and lowest in face-to-face instructional 
settings (Mean = 0.60) where its presence is not required. Network-based 
technologies (e.g., Internet, Web-based, CMC) produced a higher impact score (Mean 
= 0.72) than straight technology integration in educational settings (Mean = 0.66), 
although this difference was considered negligible. Interestingly, among the 
Pedagogical Uses of Technology, student applications (i.e., students using 
technology) and communication applications (both Mean = 0.78) had a higher impact 
score than instructional or informative uses (Mean = 0.63). This result suggests that 
the student manipulation of technology in achieving educational goals is preferable to 
teacher manipulation of technology. 

In terms of predictor variables (professional training, course design, infrastructure/ 
logistics, type of learners [general population, special needs, gifted], gender issues 
and ethnicity/race/religion/aboriginal, location, school setting, context of technology 
use, type of tool used and pedagogical function of technology) we found the 
following: professional development was underrepresented compared to issues of 



course design and infrastructure/ logistics; most attention is devoted to general 
population students, with little representation of special needs, the gifted students, 
issues of gender or ethnic/race/religious/aboriginal status; the greatest attention is 
paid to technology use in distance education and the least attention paid to the 
newly emerging area of hybrid/blended learning; the most attention is paid to 
networked technologies such as the Internet, the WWW and CMC and the least paid 
to virtual reality and simulations. Using technology for instruction and using 
technology for communication are the two highest categories of pedagogical use. 

In the final stage, the primary e-learning studies from the Canadian context that 
could be summarized quantitatively were identified. We examined 152 studies and 
found a total of 7 that were truly experimental (i.e., random assignment with 
treatment and control groups) and 10 that were quasi-experimental (i.e., not 
randomized but possessing a pretest and a posttest). For these studies we extracted 
29 effect sizes or standardized mean differences, which were included in the 
composite measure. 


The mean effect size was +0.117, a small positive effect. Approximately 54% of the 
e-learning participants performed at or above the mean of the control participants 
(50 th percentile), an advantage of 4%. However, the heterogeneity analysis was 
significant, indicating that the effect sizes were widely dispersed. It is clearly not the 
case that e-learning is always the superior condition for educational impact. 

Overall, we know that research in e-learning has not been a Canadian priority; the 
culture of educational technology research, as distinct from development, has not 
taken on great import. In addition, there appears to have been a disproportionate 
emphasis on qualitative research in the Canadian e-learning research culture. We 
noted that there are gaps in areas of research related to early childhood education 
and adult education. Finally, we believe that more emphasis must be placed on 
implementing longitudinal research, whether qualitative or quantitative (preferably a 
mixture of the two), and that all development efforts be accompanied by strong 
evaluation components that focus on learning impact. It is a shame to attempt 
innovation and not be able to tell why it works or doesn't work. In this sense, the 
finest laboratories for e-learning research are the institutions in which it is being 
applied. 


implemented appropriately, technology tools are beneficial to students' learning, and may facilitate 
the development of higher order thinking skills. 

Student manipulation of technology in achieving the goals of education is preferable to teacher 
manipulation of technology. 

Teachers need to be aware of differences between instructional design for e-learning as compared to 
traditional face-to-face situations. 

Immediate, extensive, and sustained support should be offered to teachers in order to make the best out 
of e-learning. 


Implications for Post-Secondary , 

Some educators suggest that e-learning has the potential to transform learning, but there is limited 
empirical research to assess the benefits. 

Post-secondary education would benefit from a Pan-Canadian plan to assess the impact of e-learning 
initiatives. 

It is important that instructional design match the goals and potential of e-learning. 

Research is needed to determine the feasibility and effectiveness of such things as learning objects and 
multimedia applications. 




Properly implemented computer mediated communication can enrich the learning environment; help 
reduce low motivation and feelings of isolation in distance learners. 

E-learning appears to be more effective in distance education, where technology use is required than in 
face-to-face instructional settings. 


■^^ctive ancf ef^icien?im^lemen?a^on of e-learning technologies represents new, and difficult, challenges 
to practitioners, researchers, and policymakers. 

The term e-learning has been used to describe many different applications of technology, which may be 
implemented in a wide variety of ways (some of which are much more beneficial than others). 

School administrators must balance the needs of all stakeholders, and the cost-benefit ratios of 
technology tools, in deciding not only which technologies to use, but also when and how to implement 
new technologies. 

Traditional methods of instructional design and school administration must be adjusted to deal with the 
demands of distance education and other contexts of technology use. 

Professional education, development, and training for educators must ensure that teachers will be 
equipped to make optimal pedagogical use of new methods. 


Resume: 


Cette etude preliminaire presente, dans ses grandes lignes, les resultats de 
recherche, les lacunes et les directions prometteuses observes en matiere 
d'apprentissage electronique depuis 2000, I'accent etant mis tout particulierement 
sur la situation au Canada. Notre recherche a porte sur un vaste eventail de sources 
et de documents, en vue d'englober I'ensemble des points de vue sur I'apprentissage 
electronique. Au total, notre base de donnees comportait 2 042 entrees. De ce 
nombre, 1 146 ont fait I'objet d'un examen, dont I'ensemble des travaux de 
recherche primaire et des analyses documentaires effectues au Canada. Notre etude 
a porte sur 726 documents : 235 faisaient etat de I'opinion du public en general; 131 
portaient sur I'opinion des gens du metier et praticiens; 88 etaient des documents 
d'orientation; 120 etaient des analyses documentaires; 152 consistaient en des 
recherches empiriques de base. Les divers points de vue ont ete catalogues selon les 
onze categories de variables suivantes : 1) source du document; 2) 
domaines/themes de I'apprentissage electronique; 3) valeur/effet; 4) type de 
resultat; 5) methodologie de la recherche; 6) domaine d'application; 7) facteurs de 
mise en oeuvre sur le plan pedagogique; 8) attitudes a priori; 9) types d'apprenants; 
10) contexte; 11) facteurs technologiques. 

Les donnees ont ete analysees selon plusieurs perspectives, notamment leur valeur 
qualitative en tant que resultats. Dans les travaux de recherche primaire, nous avons 
examine les methodologies de recherche utilisees. De cet examen, il est ressorti que 
plus de la moitie des etudes menees au Canada sont de nature qualitative, tandis 
que le reste est divise en deux parties d'egale importance : sondages et etudes 
quantitatives (correlationnelles et experimentales). L'analyse de la nature des 
methodologies de recherche a revele que 51 % etaient des etudes de cas 
qualitatives et 15,8 % des etudes experimentales ou quasi-experimentales. II 
semblerait done que les etudes susceptibles de nous aider a mieux comprendre « ce 
qui fonctionne » dans un contexte d'apprentissage electronique sont sous- 
representees dans la recherche faite au Canada. 

Chacun des documents a regu un code afin de faciliter I'extraction des resultats de 
I'apprentissage electronique (aussi appeles « effets » de I'apprentissage 
electronique). Les resultats/effets sont les avantages pergus ou mesures de 



I'apprentissage electronique. De leur cote, les variables explicatives sont ies 
conditions ou les caracteristiques de I'apprentissage electronique susceptibles 
d'influer sur les resultats/effets. Les effets ont ete codes selon une echelle allant du 
positif au negatif, comprenant les elements suivants : 1) realisation; 

2) motivation/satisfaction; 3) interactivite/communication; 4) reponse aux 
demandes sociales; 5) maintien aux etudes/perte d'effectifs; 6) souplesse 
d'apprentissage; 7) cout. Suivant une analyse des correlations entre ces effets, nous 
avons ensuite reduit ces categories (sauf les couts) a une seule echelle d'effets 
allant de -1 a +1. II en est generalement ressorti que la perception des effets ou leur 
mesure reelle varient d'un type de documents a I'autre. Leurs valeurs semblent etre 
inferieures dans les documents exprimant I'opinion generale, dans ceux provenant 
des praticiens et dans les rapports portant sur I'etablissement de politiques, que 
dans les etudes universitaires et les recherches primaires. Meme si, d'un cote, cela 
exprime I'espoir d'effets positifs, cela traduit peut-etre aussi, de I'autre, la realite. La 
ou nous disposions de suffisamment de documents a analyser et a coder, la valeur 
des effets s'est revelee superieure dans chacun des domaines des themes du CCA. 
Les valeurs relatives au domaine de la sante et de I'apprentissage etaient les plus 
elevees, affichant une moyenne de 0,80, tandis que celles relatives a I'apprentissage 
de niveau « elementaire/secondaire » etaient les plus basses, affichant une moyenne 
de 0,77. II n'existe cependant pas de difference significative entre ces moyennes. 

Les valeurs attributes aux effets de I'apprentissage electronique et de I'usage des 
technologies etaient plus elevees en education a distance, la ou I'apprentissage 
electronique est requis (moyenne = 0,80) que dans les contextes d'enseignement 
presentiel, la ou I'apprentissage electronique n'est pas requis (moyenne = 0,60). Les 
technologies fondees sur des reseaux (p.ex., Internet, le Web, la communication 
assistee par ordinateur) affichaient des valeurs plus elevees (moyenne = 0,72) que 
les technologies d'integration ordinaires dans les milieux educatifs (moyenne = 
0,66), meme si cette difference peut etre consideree comme negligeable. Element 
interessant : parmi les applications pedagogiques des technologies, celles faites par 
des eleves (c.-a-d. I'utilisation des technologies par les eleves) et les applications de 
communication (avec une moyenne, dans les deux cas, de 0,78) enregistraient des 
valeurs plus elevees que les utilisations educatives et informatives (moyenne = 
0,63). Ces resultats suggerent que le maniement des technologies par les eleves 
pour I'atteinte de leurs objectifs educatifs est preferable au maniement des 
technologies par le personnel enseignant. 

En ce qui concerne les variables explicatives (formation professionnelle, conception 
du cours, infrastructure/logistique, type d'apprenants [population generale, besoins 
speciaux, individus doues], variables associees au sexe, a I'ethnicite, a la race, a la 
confession religieuse et au statut d'autochtone, situation geographique, milieu 
scolaire, contexte entourant I'utilisation des technologies, types d'outils utilises et 
fonction pedagogique des technologies), les faits suivants sont ressortis : les 
donnees sur le perfectionnement professionnel ont ete sous-representees par rapport 
aux questions liees a la conception des cours et a I'infrastructure/logistique. On a 
accorde plus d'attention a la population etudiante generale qu'aux personnes ayant 
des besoins speciaux, aux eleves doues, ou aux questions liees au sexe, a I'ethnicite, 
a la race, a la confession religieuse et au statut d'autochtone. On a accorde plus 



d'attention a I'utilisation des technologies a des fins de formation a distance qu'au 
domaine emergent de I'apprentissage hybride/combine. On a accorde plus d'attention 
aux technologies basees sur des reseaux, comme Internet, le Web et la 
communication assistee par ordinateur, qu'a la realite virtuelle et aux simulations. 

Enfin, les deux categories d'usage pedagogique des technologies les plus populaires 
sont I'utilisation a des fins de formation et de communication. 

Nous avons determine les etudes primaires sur I'apprentissage electronique menees 
dans le contexte canadien qui etaient susceptibles d'etre resumees 
quantitativement. Parmi les 152 etudes examinees, 7 ont ete jugees veritablement 
experimentales (randomisation avec groupe experimental et groupe temoin), et 10 
quasi experimentales (pas de randomisation, mais tenue d'un pretest et d'un post¬ 
test). Pour ces etudes, nous avons calcule 29 ampleurs de I'effet ou differences 
moyennes normalisees, qui ont ete integrees a la mesure composee. 

L'ampleur moyenne de I'effet a ete evaluee a + 0,117, ce qui est peu eleve. Environ 
54 % des participants a I'apprentissage electronique ont fait aussi bien, sinon mieux, 
que les membres du groupe temoin (50 e percentile), un avantage de 4 %. 
Cependant, la valeur de I'heterogeneite est importante, ce qui revele une grande 
dispersion des ampleurs de I'effet. II est done clair que I'apprentissage electronique 
n'est pas toujours une condition qui maximise I'impact sur I'education. 

Dans I'ensemble, il est clair que, au Canada, la recherche sur I'apprentissage 
electronique n'est pas une priorite. La recherche sur les technologies educatives, a 
distinguer de celle sur le developpement technologique, a ete laissee pour compte. 

De plus, il semblerait que, dans le milieu canadien des recherches sur I'apprentissage 
electronique, on se soit concentre de fagon disproportionnee sur la recherche 
qualitative. En effet, nous avons constate des lacunes dans les domaines de la 
recherche portant sur I'education de la petite enfance et I'education des adultes. 

Enfin, nous sommes d'avis qu'une plus grande importance doit etre accordee a la 
mise en oeuvre de recherches longitudinales, de type qualitatif ou quantitatif (de 
preference une combinaison des deux), et que les efforts de developpement doivent 
etre etayes de composants devaluation solides mettant I'accent sur les effets de 
I'apprentissage. II serait en effet dommage de stimuler I'innovation sans savoir 
pourquoi cela fonctionne ou ne fonctionne pas. Vus sous cet angle, les meilleurs 
laboratoires d'apprentissage electronique sont les etablissements ou I'on met en 
pratique cette demarche. 

Implications pour les praticiens del a maternelle au seepndaire 5 (12 e, an nee) 

Pourvu qu ils scrient mis en oeuvre correctemenf, les outiTstecnnoTogiques sortt oenefiques pour 

I'apprentissage des eleves. Ils peuvent faciliter le developpement d'une plus grande capacite de 
raisonnement. 

Pour I'atteinte des objectifs de formation, il est preferable que les technologies soient maniees par les 
eleves plutot que par le personnel enseignant. 

Le personnel enseignant doit prendre conscience des differences de strategie pedagogique entre 
I'apprentissage electronique et les situations d'enseignement presentiel traditionnel. 

Le personnel enseignant a besoin d'un soutien immediat, ferme et resolu afin de faire de I'apprentissage 
electronique un franc succes. 

Implications pour I'education postsecondaire , , 

Ceniains educateurs avancent que l apprentissage electronique pourrait theoriquement transformer les 
precedes d'apprentissage, mais peu de travaux de recherche empirique permettent d'en evaluer les 
avantages. 



• L'education postsecondaire gagnerait a mettre en place un programme pancanadien devaluation de I'effet 
des programmes d'apprentissage electronique. 

• II importe que la conception pedagogique s'harmonise avec les objectifs et le potentiel de I'apprentissage 
electronique. 

• Des travaux de recherche sont necessaires afin de determiner la praticabilite et I'efficacite des objets 
d'apprentissage et des applications multimedias, entre autres. 

• Des systemes de communication assistee par ordinateur correctement mis en oeuvre peuvent enrichir 
I'environnement d'apprentissage ainsi que remedier au faible taux de motivation et au sentiment 
d'isolement chez les apprenants a distance. 

• L'apprentissage electronique semble plus efficace dans un contexte de formation a distance requerant le 
recours aux technologies que dans un contexte d'enseignement presentiel. 

• ‘t^ 7 ^ifse a en°5u\/re > eFfK^ce^eff^nc(f?nnelle des technologies d'apprentissage electronique lance de 
nouveaux defis complexes aux praticiens, aux chercheurs et aux decideurs. 

• Le terme « apprentissage electronique » est utilise pourdecrire une multitude d'applications 
technologiques qui peuvent etre mises en oeuvre de differentes manieres (certaines etant plus 
avantageuses que d'autres). 

• Les responsables scolaires doivent concilier les besoins de tous les intervenants et le rapport couts- 
avantages des outils technologiques, au moment de decider non seulement du type de technologies a 
mettre en place, mais aussi du moment et de la fagon de les mettre en place. 

• Les methodes traditionnelles de conception pedagogique et d'administration scolaire doivent etre adaptees 
de fagon a pouvoir composer avec les exigences de l'education a distance et d'autres contextes 
d'utilisation des technologies. 

• La formation et le perfectionnement professionnels des educateurs doivent garantir que le personnel 
enseignant aura le bagage necessaire pour tirer le meilleur profit de la mise en pratique de ces nouvelles 
methodes sur le plan pedagogique. 

A review of e-learning in Canada: A rough sketch of the evidence, 
gaps and promising directions: Introduction 


E-learning has become the general term encompassing the application of computer technologies to 
education, whether it occurs in face-to-face classrooms, in blended and hybrid courses, in mediated 
distance education contexts or in online learning environments. The Canadian Council on Learning 
(CCL) defines e-learning as the development of knowledge and skills through the use of information 
and communication technologies (ICTs) particularly to support interactions for learning- 
interactions with content, with learning activities and tools, and with other people. It is not merely 
content-related, not limited to a particular technology and can be a component of blended or hybrid 
learning (Rossiter, 2002, also 2005 in an address at the CCL Workshop on E-Learning). The CCL 
recognizes the breadth of the concept of learning through electronic means, as well as its growing 
pervasiveness in Canadian institutions of formal learning, at the level of elementary and secondary 
schools and of colleges, and universities, as well as in early childhood and health-related learning. 


Enthusiasm for, as well as apprehension regarding, the use of e-learning appears widespread as we 
herald the arrival of the Information Age. To some, e-learning can be used as a powerful and 
flexible tool for learning (Hannafin & Land, 1997; Harasim, Hiltz, Teles &Turoff, 1995; Lou, Abrami &. 
d'Apollonia, 2001; Scardamalia &. Bereiter, 1996). Proponents of the application of electronic 
technologies to education have long argued that computers possess the potential to transform 
learning environments and improve the quality of the learning that results (e.g., Bransford, Brown &. 
Cocking, 2000; Jonassen, Howland, Moor & Marra, 2003; Kuh &. Vesper, 2001; McCombs, 2000, 
2001; Siemens, 2005; WBEC, 2000). Possible means include: increasing access to information 
(Bransford et al. 2000); providing access to a richer learning environment (Bagui, 1998; Brown, 
2000; Caplan, 2005; Craig, 2001); making learning more situated (Bransford et al. 2000); 
increasing opportunities for active learning and inter-connectivity (Laurillard, 2002; Shuell &.Farber, 
2001; Yazon, Mayer-Smith &. Redfield, 2002); enhancing student motivation to learn (Abrami, 




2001); and increasing opportunities for feedback (Jonassen et al. 2003; Laurillard, 2002). A 
recently completed study (Lowerison, Sclater, Schmid &. Abrami, 2006) points to the role of 
technology as a catalyst for change, supporting the learning process through course design and 
motivation. Indeed, there is sufficient optimism for technology's positive impact that governments 
have established committees, formed task forces, and dedicated substantial funds to the delivery 
or enhancement of technology-based instruction (CMEC, 2001). 

There has also been scepticism (Clark & Sugrue, 1995; Cuban, Kirkpatrick & Peck, 2001; Healy, 
1998; Noble, Sheiderman, Herman, Agre, & Denning, 1998; Russell, 1999) about the use of 
technology to improve learning, including suggestions that it represents a threat to formal 
education, from kindergarten through university. For example, it may create an imbalance between 
computer skills and essential academic and thinking skills, foster technology dependencies and 
isolation rather than independent and interdependent learners, and/or erode the joy and motivation 
to learn, replacing it with frustration because of failed equipment, etc. Some teachers hold beliefs 
concerning the usefulness of information and communication technologies (ICT) that parallel their 
attitudes towards any change to teaching and learning, be it through government mandated reform 
or societal pressure. "If the computer can accomplish the task better than other materials or 
experiences, we will use it. If it doesn't clearly do the job better, we will save the money and use 
methods that have already proven their worth" (Healy, 1998, p. 218). Even moderate 
commentators such as Clark and Sugrue point out that the most likely explanation for increased 
learning with computer technology is instructional method differences, content differences, or 
novelty effects, and not the technology itself. 

On June 6, 2005, MacLean's, Canada's weekly newsmagazine, featured an article on e-learning. 
Emblazoned across the cover was the title: "How computers make our kids stupid." The article was 
one reporter's attempt to weave together personal anecdotes, stories, and observations with a 
smattering of selective research evidence. The tale she told was compelling and generated instant 
and widespread reaction. Certainly the expose resulted in the publisher's goal—selling magazines. 
But what about the evidence used in the report? Was it accurate and complete? Did it provide a 
balanced portrait of the evidence, the important questions, the known, and the unknown? It 
certainly did not. 

Researchers face a similar challenge when they have to sift through the accumulated evidence to 
form an accurate picture about scientific knowledge. Results vary, often from extremely positive to 
extremely negative, and it is a challenging task to synthesize complex and detailed evidence 
succinctly, accurately, and meaningfully. Increasingly, this same challenge exists for policy-makers, 
practitioners, and the public in a multitude of areas related to the social sciences and education. 

Several years ago Ungerleider and Burns (2002), reviewing mostly Canadian research, found little 
methodologically rigorous evidence of the effectiveness of e-learning in promoting achievement, 
motivation, and metacognitive learning, or on facilitating instruction in content areas in elementary 
and secondary schools. Ungerleider and Burns also emphasized that student academic achievement 
does not improve simply as a result of having access to computers in the classroom without 
concurrent changes to instruction. 

Does the application of electronic technology, under the general rubric of e-learning, represent a 
positive development in formal educational settings that should be embraced and pursued, or is it a 
danger to the achievement of the general societal goal of an educated and skilled populace? Does it 
actually "make our kids stupid"? There is little doubt about the ongoing and future need for a 





population that is skilled in the use of computers, that is, learning about technology—but what 
about learning through technology? The answer to this question is complex, partly because of the 
vast array of technologies (both hardware and software) that are available for e-learning and their 
many uses. However, this complexity is magnified when one considers the variety of contexts, 
learners, subject matters, etc. that make up the fabric of formal learning in Canadian schools and 
universities. 

Our review is an up-to-date, comprehensive examination of e-learning with a special emphasis on 
Canadian research. Because Canada is investing heavily in the uses of technology for learning, both 
in classrooms across the nation and at a distance, this project will specifically address the need for 
up-to-date and comprehensive answers to practical questions that are required by decision-makers 
at the local, provincial and national levels, and by practitioners who are involved in the day-to-day 
delivery of quality educational experiences to Canada's next generation of informed citizens. 

We will address three objectives: 1) identify the most effective practices and procedures to 
promote learning; 2) identify major gaps in our knowledge and understanding of e-learning; and 3) 
identify the most promising lines of inquiry for addressing those gaps. Our review will be different 
than a traditional review of primary research. Instead, our review will be much broader and 
incorporate arguments from a number of sources. Such a catalogue of arguments will provide a 
comprehensive picture of the state of a field from the perspectives of research, policy and 
practitioner orientations. It will allow an exploration of what works (best practices), an assessment 
of what is incomplete in the literature, and an informed vision of promising lines of new research. 
Therefore, the expected outcomes of the proposed literature review include: 1) a description of the 
major issues facing Canadian educators and policy-makers in regards to the application of e- 
learning in educational and health institutions and as an avenue for preparing children and adults 
for lifelong learning; 2) an assessment of the effectiveness of the technologies being harnessed for 
educational purposes; and 3) identification of the factors that both enhance and detract from the 
effects of e-learning. We will explore the following sub-areas and themes: Adult Learning; Early 
Childhood Learning; Elementary and Secondary School Learning; Health and Learning; and 
Postsecondary Learning. 

Approaches and Methodology 

This literature review consists of a systematic compilation of current public, policy-making, 
practitioner and scholarly opinions and positions on the topic of e-learning in the five theme areas. 
In other words, it goes beyond the usual review of research, limited to primary evidence, by 
including multiple sources of information, to insure that a broad base of arguments and opinions 
about e-learning is represented, and to help situate the primary evidence and the scholarly reviews 
of the evidence. It also includes, therefore, an analysis of news media, practitioner opinions, and 
policy documents. We call this compilation of evidence and opinions from a variety of sources an 
"Argument Catalogue." The Argument Catalogue codebook provides categories and values that can 
be applied across both constituencies and theme areas in order to classify conjectures, opinions, 
concerns, and expert judgments, The Argument Catalogue helps frame the questions and inform the 
analyses in order to know what generalizations may be drawn from the existing empirical evidence, 
what gaps exist, and what may be fruitful avenues for further research. 

The value of an Argument Catalogue includes being able to identify the consistencies and 
inconsistencies that exist between research evidence, practitioner experience, and public 
perception. For example, do different sources agree about the impact and value of e-learning? 



Identifying similarities and differences among these sources may help us understand better how to 
mobilize knowledge to impact policy and practice. Another value of an Argument Catalogue is 
developing a comprehensive understanding of the issues surrounding e-learning including its 
impacts, applications, and the pedagogical, technical, and situational factors which moderate its 
effectiveness and efficiency. The combination of multiple sources of expertise and evidence- 
scholarly evidence and understanding from the field—may strengthen or qualify conclusions from 
any single source and help identify gaps in understanding, offering promising new directions for 
research. In other words, an Argument Catalogue attempts to provide a comprehensive and 
inclusive framework for understanding e-learning by giving voice to all the key constituencies who 
generate and apply what has been learned. 

The standards which we strove to achieve in this review included: being comprehensive and current 
about the documents we reviewed; undertaking a review which was repeatable by others and used 
a transparent review methodology; insuring the outcome of the review was clear, the findings were 
useful, and the results were credible; and finally, using a methodology that was precise, so that our 
conclusions would be accurate. We sought to be comprehensive and current in order to insure that 
we had included the widest range of evidence available to date. To meet this objective required 
that we search, retrieve and code a large body of evidence. We especially wanted to avoid drawing 
final conclusions from an insufficient and unrepresentative collection of evidence. We sought to use 
methods that were repeatable and transparent by outlining each of the steps we followed in the 
review, so that other reviewers could replicate our methods. We also wanted the review to be clear 
and useful; we hope that its inclusive nature helps to insure that our readers find the results to be 
understandable and credible. Finally, we wanted our review to be precise and accurate; however, 
time constraints and the scope of the review have limited the extent to which we achieved this 
objective. The breadth of the review conflicted with the extent to which we could explore 
documents in depth during the time available for the compilation of this report. Accordingly, we 
have called this a rough sketch of the state of the field. 

Locating the Documents 

A systematic search of the literature was conducted on current public, policy-making, practitioner 
and scientific opinions and positions related to the topic of e-learning in the following theme areas: 
Adult Learning; Early Childhood Learning; Elementary and Secondary Learning; Health and 
Learning; and Postsecondary Learning. Specifically, literature from the last five years (2000-2005) 
for the following areas was located: the policy and decision makers' perspective (e.g., policy 
papers, grey literature); public perception (e.g., national and provincial media); practitioners' 
perspectives (trade periodicals); theoretical and empirical reviews by scholars; and primary studies 
conducted in Canadian contexts. 

In the first four instances, emphasis was placed on locating French and English Canadian literature, 
although international sources were also consulted. In the last instance, only primary studies 
(quantitative and qualitative) on e-learning that were conducted in Canada were located. 

Retrieval tools used : The following databases and web-based sources were consulted to locate 
relevant items within each sector. 

• Policy: AACE Digital Library, CPI.Q, ERIC, FRANCIS, Google, Source OECD, individual websites 

• Public (Media): Biblio Branchee, Canadian Newsstand, CBCA Current Events, CPI.Q 

• Practitioner (Trade): AACE Digital Library, CBCA Current Events, CBCA Education, CPI.Q, EBSCO Academic 
Search Premier, ERIC, individual websites 

• Scholarly articles and reviews: AACE Digital Library, CBCA Education, Cochrane Library, EPPI, ERIC, 



EBSCO Academic Search Premier, FRANCIS, Google Scholar, Medline, ProQuest Digital Dissertations and 
Theses Full-text, PsycINFO. 

Additionally, manual searches of selected journals were performed: Canadian Journal of Learning 
and Technology, Journal of Distance Education, and the International Review of Research in Open 
and Distance Learning. Finally, branching (scanning of bibliographic references at the end of 
selected articles) was used. 

Strategy ; Search terms were selected from the following definition of e-learning: "E-learning is 
the development of knowledge and skills through the use of information and communication 
technologies (ICTs) to support interactions for learning—interactions with content, with learning 
activities and tools, and with other people" (Rossiter, 2002; 2005). 

A variety of broad terms related to the use of technology in education were used. These included 
electronic learning, distance education, distance learning, online instruction, multimedia instruction, 
online courses, web-based learning, virtual classrooms, computer mediated communication, 
computer-based instruction, computer-assisted instruction, "technology uses in education," 
telemedicine, and e-health. 

Specific applications of technology (e.g., learning objects, digital portfolios, specific instructional 
software, etc.) were not used as keywords or descriptors within individual searches, as this would 
have significantly widened the scope of retrieval. Note, however, that in some cases retrieved 
items did describe specific applications of technology. 

Controlled vocabulary terms were used whenever possible, thus the actual terms used varied 
according to the field (education, psychology, medicine) and sector (e.g., media vs. scholarly) 
searched and the controlled vocabulary used within a particular retrieval tool. Please consult the 
CSLP website at http://doe.concordia.ca/cslp/CanKnow/eLearning.php for a complete list of the 
search tools consulted, along with the original search strategies for each. 

Results : Overall, there were 2,042 articles in our database identified through the searches. 
Because of time limitations, of these we reviewed 1,146, including all the Canadian primary 
research and all scholarly reviews of the literature that were retrieved. Of these, 726 documents 
were included for further analyses: 

• 235 - general public opinion 

• 131 - trade practitioners' opinion 

• 88 - policy documents 

• 120 - reviews 

152 - primary empirical research The 420 excluded documents were rejected for the following 
reasons: 

• "Not e-learning" - 198 

• "Workplace learning" - 43 

• "Other (specified) reasons" - 51 

• "Primary research conducted outside of Canada" - 128 

The CSLP website at http://doe.concordia.ca/cslp/CanKnow/eLearning.php provides a complete list 
of the 726 citations. 

Coding the Documents 

An Argument Catalogue codebook was developed to help the reviewers document our research 



process; this document was used as the basis for coding the articles. The codebook can be found at 
the CSLP website athttp://doe.concordia. ca/csl p/Can Know/e Lea rnina.php 


The Argument Catalogue codebook was developed by taking a sample of documents of various 
types to insure that the major issues covered by the documents were reflected in the codebook. 
The intent was to insure that the codebook reflected the important topics raised in the different 
literatures. A combination of closed-ended and open-ended (i.e., emergent) coding was employed. 
The documents from each of the five theme areas were subsequently coded using the codebook. 

The Argument Catalogue codebook includes the following classes of variables (the list below is not 
exhaustive but deals with those categories on which the most emphasis was placed in the 
analyses): 

1. Document Source (i.e., general public opinion, trade/practitioner's position, policy-making reports, 
scholarly/academic reviews, and primary empirical research) 

2. Areas/Themes of e-learning (i.e., Adult Learning, Early Childhood Learning, Elementary and Secondary 
School Learning, Postsecondary Learning, Health and Learning) 

3. Value/Impact 

4. Achievement 

5. Motivation/satisfaction/attitudes 

6. Interactivity/communication 

7. Meeting social demands 

8. Attrition/retention 

9. Learning flexibility/accessibility 

10. Cost 

11. Type of evidence (i.e., opinion, survey, quantitative data, qualitative data 

12. Research design (i.e., p re-ex peri mental, correlational, quasi-experimental, true experiment/RCT) 

13. Area of applicability (i.e., International, American, Canadian, provincial, local) 

14. Pedagogical implementation factors such as: 

° The provision of professional training 
o Course development practices 
o Logistical and infrastructure support 

15. A-priori attitudes (not analysed; insufficient cases) 

o Parents 
o Teachers 
o Students 
o General Public 
o Administrators/institutional 
° Authors 

16. Types of learners 

■ Special needs 

■ General 

■ Gifted 

■ Gender 

■ Race/native/religious/ethnicity 

17. Context 

■ Location (i.e., rural, urban, inner city, suburban) 

■ School type (i.e., public, private, home schooling) 

18. Technology Factors 

■ Context of technology use (i.e., distance education, in class, blended, unspecified) 

■ Type of tools used (i.e., internet/intranet/on-line/web, virtual reality/learning 
objects/simulations, technology integration—computers and software for particular purposes, 
unspecified) 

■ Intervention type, including: Instructional (e.g., drill, practice, tutorials, remediation); 
Communicative (e.g., e-mail, ICQ, computer conferencing, LCD projector); Organizational (e.g., 
data base, spreadsheets, record keeping, lesson plans); Analytical/Programming (e.g.. 



statistics, charting, graphing, drafting, robotics); Recreational (e.g., games); Expansive (e.g., 
simulations, experiments, exploratory environments, brainstorming); Creative (e.g., desktop 
publishing, digital video, digital camera, scanners, graphics); Expressive (e.g., word 
processing, on-line journal); Evaluative (e.g., assignments, portfolio, testing); Informative 
(e.g., Internet, CD-ROM); and Unspecified/Missing. 

The coding of the documents was done by a team of eight graduate students, most of whom had 
previous experience coding studies for other systematic reviews. The team met weekly with the 
faculty supervisors to review progress and to address questions and concerns. Because of the 
volume of the documents to be coded, and because of the limited timeframe, no article was coded 
by more than one student; therefore no measure of inter-rater reliability could be calculated. To 
compensate for this, preliminary meetings were conducted in order to reach common 
understandings of the meanings of the coding categories and values; several [5-10] documents 
from each category were coded by all team members, and differences were resolved through 
discussion. 

Data Entry and Analysis 

Each coder entered data into a Microsoft Excel spreadsheet that included both open-ended and 
closed-ended category values. The separate Excel files were combined to form a master database, 
which was then converted to an SPSS data file. The data were spot checked for accuracy and 
corrected prior to the main analyses. 

Results 

The Results are divided into five sections: Sample of Document Conclusions; Analysis of the 
Documents Retrieved and Coded; Frequencies of Predictor Variables and Impacts of E-Learning; 
What Predictor Variables Explain Technology Impacts? and Quantitative Summary of the Canadian 
Primary Research on E-learning. 

Sample Of Document Conclusions 

The team reviewed all open-ended (emergent) coding intended to identify, when possible, the 
authors' principal positions, arguments presented, or conclusions reached, in every document. 
Mirroring the procedure for coding in general, preliminary meetings were conducted to assure the 
common understanding of the task, after which one coder concentrated on the analyses of each 
type of literature source. The most salient, interesting, informative, or powerful message or the 
most representative or frequently appearing message in each document was extracted. These 
summaries appear below organized by the source of evidence. 

General public opinion: 

One of the major messages found in the general public opinion literature is that e-learning is a 
rapidly growing field in education. There is speculation that traditional Canadian universities will 
come under increased pressure to expand their online offerings because of growing enrolments 
("Online Courses Grow in Popularity", 2003). Online courses, and institutions such as Athabasca, 
are viewed as possible solutions to the problems of growing enrolments and limited physical space 
on traditional campuses (Amiot, 2001; Rodrigue, 2001; Sinnema, 2005; Summerfield, 2000; 
Wanless, 2000). 

Similarly, there is the perception that e-learning can provide greater access to educational 
programs for individuals who otherwise would not be able to participate in them. Computer-based 
distance education is viewed as a means for learners restricted by time or distance to engage in 
continuing education. In Timiskaming, elder Jenny Enair's opinion, "For anyone who is motivated to 



learn, but must care for an older parent or young children at home, distance learning is truly a 
godsend," ("Contact North Enjoys High Enrolment in Tri-Towns", 2002)—a position shared by many 
others (Foster, 2002; Hamilton, 2002; "Online Learning Opens Up Options: Computers Bring School 
to Busy Lives", 2003). 

There are, however, two prevalent concerns about e-learning. The first is its high costs and the 
potential risk that money will be diverted away from more appropriate or urgent projects to fund e- 
learning. One article in the Toronto Star informed readers that, despite the potential benefits of 
having education available to all Canadians, the cost of establishing nation-wide high-speed 
internet access could cost four billion dollars ("Internet for All", 2001). Another reported on the fear 
that computer-based work teaches skills that may become obsolete at the expense of time-tested 
skills, such as critical thinking ("Classroom Computers Get Failing Grade: Fixing High-Tech Tools 
Keeps Teachers From Their Primary Tasks", 2000; "Technology v Textbooks: Costly Computer 
Systems Compromise Other Essential Teaching Needs", 2001; Thibodeau, 2001). 

The second concern is that e-learning might have a negative impact on the development of 
children's creative skills. Some educators, while seeing benefits to the use of technology by young 
children, caution that such use must be guided. "It has to do with how you use a powerful tool ... 
[you] can't just sit them in front of a computer and say: go to it" one expert is quoted as saying 
("Computers and the Young: They Can Impede Development, One Group of Experts in the U.S. 
Says. Meanwhile, Quebec Is Forging Ahead With Its New-Technology School Curriculum", 2000). 
This concern can also be found expressed in "The Birth of the Celtic Tiger" (2003), and Kelly, Lord 
and Marcus (2000, September 25). 

Finally, there is the belief that, despite the growth of distance education and other forms of 
technology-based learning, teachers and classrooms will remain a necessary part of the education 
system. Teachers' ability to communicate with students and to facilitate their learning is still highly 
valued (Dibbon, 2000; Wake, 2000; Wood, 2000). 

Practitioners' point of view: 

Practitioners are mostly positive about the development of E-learning. They believe that e-learning 
brings more opportunities and flexibility to learning. Advanced technologies play an important role 
in reaching out to and connecting with students who live in remote areas, who are homebound, and 
who have the demand for life-long learning (Crichton & Kinsel, 2000; Haddow, 2003; Hillman, 2000; 
Johnston & Mitchell, 2000; Oakley &. Stevens, 2000). Another message is that e-learning, to some 
extent, enhances self-regulated learning (Hadwin &Winne, 2001). 

One of the major concerns emerging from trade and practitioners' points of view is that e-learning 
requires careful attention to instructional design and planning (Bell, McCoy, & Peters, 2002; 
Hofmann & Dunkling, 2002; McKenzie, 2002; Nelson, 2001;). Technologies provide lots of learning 
potential but they do not guarantee learning. As Peter Swanneell, the Vice-Chancellor of the 
University of Southern Queensland stated, "Without a new pedagogy, the technology will fail" (cited 
by Hofmann &. Dunkling, 2002). We need a proper instructional plan before we integrate technology 
in the class. However, some educators note that there are not enough financial means for providing 
such instructional support. Although professional training and support is crucial for the success of e- 
learning, they may suffer from competition with e-learning technologies for financing (Lewis & 
Jenson, 2001; Tuinman, 2000). In a way, the emergent technologies diverting substantial financial 
resources from other components of educational infrastructure force us to reconsider our priorities. 
Too much funding goes towards the purchase, installation, and maintenance of technology per se 



and less for teacher support and professional training. The message delivered is very clear: we 
need to invest more on the pedagogical side of the integration of technology in classrooms. 

The development of e-learning requires universities to have new policies and strategies to meet the 
emerging social demand brought by advanced technologies (Fielden, 2001; Shapiro, 2002). The 
implementation of new technologies exposes us to a global market, makes us collaborate with our 
competitors, and challenges the traditional monopoly over the accreditation and certification, while 
bringing to the picture new sources of revenue. The change starts from the integration of 
technology in classroom and spreads to the whole education system. 

Policy documents: 

The overall perceptions of technology use in education by the policy making community are 
favourable. The HORIZON Report (New Media Consortium, 2004) contains prognoses of what 
technologies will become important in higher education in the next one to five years. Several 
papers emphasize the role Canada should assume in implementing technology-based approaches to 
education (Industry Canada, Advisory Committee for Online Learning, 2001; Rossiter, 2002) and 
why it is both possible and necessary. Finally, Davis and Carlsen (2004) named four major reasons 
to invest in ICT in education: 1) ICT is an imperative for economic competitiveness; 2) ICT 
increases educational attainment; 3) ICT increases access to education; and 4) ICT is a catalyst 
promoting changes in education. 

One of the major messages from the policy documents is about the need to bridge the gap between 
theory, research and practice (Kinuthia, 2004; Muhlhauser, 2004; Pittard, 2004). There is a 
concern, however, about how exactly it is done. According to Oliver and Conole (2003), the use of 
evidence-based practice in policy making can be criticized on methodological, epistemological, and 
moral grounds: "While the aim of evidence-based practice—linking research, practice and policy—is 
valuable, its methods are questionable" (p. 394). Among the problems with this ideology they 
mentioned "inappropriate dismissal of qualitative approaches," and "disempowering practitioners" 
by forcing them into a tradition "that blends evaluation, research and practice together" (p. 394). 

Policy documents have also generated a substantial number of "conditional" messages, specifying 
what conditions should be met, what factors taken into account, and what approaches implemented 
in order to make e-learning really efficient. Harley (2002) emphasized the need to know when and 
how to get involved in distance education. Merisotis (2001) named the following benchmarks for 
success in Internet-based distance education: the need for learning outcomes—not the availability 
of existing technology—to be the determinates of whether and how the technology should be used 
to deliver course content; the necessity of preliminary information and technical assistance for 
students and faculty to facilitate their way into and through technology-based courses; properly 
documented and accessible course standards and requirements, as well as security measures; 
account for individual differences in learners personalities and motivation, etc. This list matches 
closely the conditions necessary for successful use of technology in teacher education produced by 
the UNESCO Division of High Education (2002): consistent policies and standards, accounting for 
cultural context, adequate infrastructure, proper professional training, attention to pedagogical 
issues, etc. See also Padilla and Zalles (2001) or Katz (2002) who insisted that the most important 
directions for policies in the area of ICT-based learning are educational (and not technological) 
access, literacy, matching budgeting with standard-setting activities, balancing access with privacy 
and customizing and personalizing service offerings. 


Literature reviews: 



Attitudes towards e-learning, reflected by scholarly and academic reviews, range from neutral to 
positive. On one hand, it is noted that e-learning (e.g., DE, CAI, etc.) is at least as effective as 
traditional instructional strategies ( Rosenberg, Grad, & Matear, 2003), and that there are no major 
differences in academic performance between the more traditional and more technology-oriented 
modes of instruction (Cavanaugh, 2001). On the other hand, many reviews go further, reflecting a 
particularly positive attitude towards the impact of e-learning (Kulik, 2003; Mayer, 2003; 
Steinberger, 2002). Benefits include offering a variety of new possibilities to learners (Breuleux, 
Laferriere, & Lamon, 2002), in addition to having a positive effect on students' achievement in 
different subject matter areas (Chambers, 2003; Christmann &. Badgett, 2003; Soe, Koki, & Chang, 
2000 ). 

However, reviews also acknowledge the need to address more closely design issues in e-learning 
courses and activities. Developing effective strategies for teaching and learning is also called for 
(Meredith &. Newton, 2004; Oliver &. Herrington, 2003). More specifically, designing instruction to 
compliment the positive attribute of computer technology is advocated by Berg (2000). Addressing 
learners' needs in the design of e-learning activities is suggested by some reviews (Crawford, 
Gannon-Cook, & Rudnicki, 2002; Ewing-Taylor, 1999). Wherever the implementation issues are 
addressed, there seems to be a consensus among reviewers that effective use of e-learning 
requires the presence of immediate, extensive, and sustained support (Knowles, 2004; Sclater, 
Sicoly, &. Grenier, 2005). 

Nevertheless, reviews report a major concern regarding the absence of strong empirical evidence 
to support the use of e-learning (Torgerson & Elbourne, 2002; Urquhart et al., 2002; Whelan & 
Plass, 2002). One review considered the quality of research to be inadequate and called for more 
scientific rigour and less reliance on anecdotal evidence (Terrell, 2002). Another review emphasized 
that advances in DE technologies are outpacing research on their effectiveness (Hirumi, 2002). An 
extra obstacle facing the advancement of research in the field seems to be the fact that e-learning 
researchers are not all uniform in the methods used and questions asked (Cantoni & Rega, 2004). 

Primary empirical studies: 

A review of the empirical research conducted in Canada revealed several fairly broad consensuses. 
One point is that some learners are better prepared than others to use e-learning technologies to 
facilitate their educational progress; individual "readiness" seems to be a crucial factor in 
accounting for the success of e-learning applications in education. Cuneo, Campbell & Harnish 
(2002) list several individual characteristics that may determine the outcomes of technological 
interventions: motivation, computer skills, literacy skills, communication skills, and learning styles. 
In a separate paper, Cuneo and Harnish (2002) point out that "quasi-open computer-mediated 
environments are not safe places for students unsure of their writing skills and knowledge ... online 
learning might not be appropriate for all students" (p. 19). Looker and Thiessen (2002), 
commenting on the "digital divide" for Canadian youth, remarked that access to, and experience 
with, computer technology determines "computer competency", and that this competency is 
generally associated with urban residents of higher economic status. Their survey of Canadian high 
school students also indicated that females demonstrated less interest (and less confidence) in 
achieving computer competency. Bryson, Petrina and Braundy (2003) studied "gender- 
differentiated participation" in British Columbia schools, noting that "the current percentage of girls 
enrolled in technology-intensive courses remains extremely low, while performance data indicate 
that those female students who participate in these courses do better, on average, than male 
students in these courses" (p. 191). Levin and Arafeh (2002) have also remarked on the differences 



between students who are "internet-savvy" and those who have had little opportunity to develop 
their experience with networking tools. Dewar and Whittington (2000) concluded that adult 
learners' learning styles (as indicated by Myers-Briggs personality types) can predict the pattern of 
their participation in online courses, while Li (2002) observed that, "female students tend to initiate 
conversations, while male students are more likely to enter the dialogue at later stages and 
respond to previous discussions" (p. 341). Individual metacognitive factors are also implicated in 
student success; Karsenti (2001) pointed to the relevance of self-direction and self-regulation in 
university students, concluding, "The main difficulty encountered by students seemed to be their 
lack of autonomy or the trouble they had in learning by themselves, in managing their own 
learning" (p. 33). 

A second conclusion is that effective instructional design for e-learning applications does not 
resemble traditional pedagogical methods of information delivery and competence assessment. 
Cuneo and Harnish (2002) remarked, "The collaborative features of FirstClass [online conferencing 
software] are compatible with deep and comprehension learning - asking questions, seeking 
understandings, exploring ideas freely" (p. 19). Schnackenberg, Luik, Nisan, and Servant (2001) 
concluded, "Many teachers are unsure of what the curriculum objectives are regarding the use of 
computers at different grade levels ... Not enough current pedagogical direction exists for teachers 
interested in integrating computer technology into the curriculum" (p. 152). Cheng and Myles 
(2003) suggest that "a change in mind-set" is needed for teachers to realize what they can do, and 
do better, with new technologies. According to Plante and Beattie's (2004) survey of Canadian 
elementary and secondary school principals, "most teachers possessed the required technical skills 
to use ICT for administrative purposes such as preparing report cards, taking attendance or 
recording grades, while fewer had the necessary qualifications to effectively engage students in 
using ICT to enhance their learning" (p. 25). Murphy (2004) describes the need for research into 
how post-graduate learners, exploring complex and ill-structured problems, can use online 
asynchronous discussion to advance their learning in collaborative environments; Murphy and 
Coleman (2004) see this type of learning environment as "a medium that allows for a shift in the 
locus of communication, interaction and control from the teacher to the learner or from the one-to- 
many, teacher-to-students' mode to a many-to-many, students-to-students' mode" (p. 41). 
Kanuka, Collett &. Caswell (2002) recommend that "risks" be taken in the design and development 
of new methods for exploiting technological tools. 

Given the above, the third consensus, which arose from the primary research, is unsurprising: 
teachers require focused professional development training to learn how to gain optimal benefits 
from e-learning technologies. Jacobsen, Clifford, and Friesen (2002) note that "questions of how to 
prepare a new generation of teachers, many of whom have been schooled in old ways despite their 
relative youth, are increasingly pressing in their urgency" (p. 1). Morgan, White, Portal, Vanyan and 
Lazenby (2002) studied the implementation by the Toronto Catholic School Board of software 
designed to promote literacy skills; they surveyed the teachers, who indicated that their training 
for the project had been inadequate to the task. A survey of public school administrators in Alberta 
( Irvine & Montgomery, 2001) revealed the collective attitude that ongoing in-service teacher 
training is a clear requirement for the Alberta school system; many respondents felt that university 
teaching instruction provided inadequate preparation in the realm of "curriculum integration." 
Schnackenberg et al. (2001) recommended that the training of teachers needs be considered in the 
early stages of planning for technology integration, claiming, "if teachers were contacted in the 
planning stages, items such as training and support would not be overlooked" (p. 159). 



Several studies agreed on a fourth conclusion that the collaborative methods afforded by online 
technologies facilitate the development of higher-order critical thinking, providing great potential 
for educative dialogues. Lapadat (2000) in particular credits the time available for reflection as 
allowing for the construction of "deeper levels of understanding" (p. 20) by university students; 
Lapadat (2004) claims that "discursive interaction in an asynchronous, text-based, online course 
may be uniquely suited to fostering higher-order thinking and social construction of meaning" (p. 
236). Similarly, Pear and Crone-Todd (2002) found that cardiac nurses used online discussions to 
construct critical educational discourses to support their practices, while Gabriel and MacDonald 
(2002) reported that their students claimed that discursive participation in an online MBA had a 
"major impact on their lives. 

A fifth message is that e-learning provides disabled students with previously unavailable 
educational opportunities. Artiss, Fitzpatrick, Hammet, Kong and Noftle (2001) reported "amazing 
success" in supporting disadvantaged adults in increasing their traditional and technological 
literacy. Fichten, Asuncion, Barile, Fossey and Robillard (2001) concluded that "the potential of 
computer and adaptive computer technologies to remove barriers for students with disabilities is 
enormous" (p. 52), and collaborative online educational processes have been shown to benefit the 
reintegration of students after traumatic brain injuries (Verburg, Borthwick, Bennet & Rumney, 
2003). 

Analysis of the Documents Retrieved and Coded 

To orient the reader, we began with a selective summary of the major conclusions from the five 
sources of evidence. We now turn our attention to a more detailed and comprehensive analysis of 
the documents. 

Table 1 presents the number of documents retrieved in each of the theme areas and Table 2 
contains the diverse sources of evidence that were included in the review. These data are shown 
graphically in Figures 1 and 2. The preponderance of the documents pertains to elementary, 
secondary, and postsecondary education. There are far fewer articles which discuss adult education 
or health education, and very few which address early childhood learning. This lack of evidence 
limits the extent to which analyses of some theme areas were performed. 


Table 1. Frequency of documents by theme area 


CCL Theme Areas 

Frequency 

Percent 

Adult Education 

2S 

3.9 

Early Childhood Education 

S 

1.1 

Elementary Secondary Education 

235 

32.4 

Postsecondary Education 

356 

49 

Health and Learning 

62 

8.5 

Not Specified 

37 

5.1 

Total 

726 

100 













Figure 1. Frequency of documents across the CCL Theme Areas 


Table 2. Frequency of documents by source 


Document Source 

Frequency 

Percent 

General Public Opinion 

235 

32.4 

Trade practitioners' position 

131 

IS 

Policy making reports 

SS 

12.1 

Scholarly academic reviews 

120 

16.5 

Primary* empirical research 

152 

20.9 

Total 

726 

100 


















Table 3 shows how all the documents included in the review were distributed by the geographic 
region of applicability of the issues they covered. The decisions whether to code for geographical 
area (scope) of applicability were made whenever an explicit statement linking issues addressed 
(applications or methodology used and/or perspectives of further development considered) to a 
particular country (province, etc.) was detected or such a connection was obvious from the 
context. 


Table 3. Frequency of documents by geographic region 


Geographical Region 

Frequency 

Percent 

International 

138 

19 

U.S.A. 

78 

10.7 

Canada 

122 

16.8 

Provincial 

123 

16.9 

Unspecified 

265 

36.5 

Total 

726 

100 


Table 4 describes the distribution of the primary Canadian research literature by theme areas. Here 
we can see that the preponderance of the research on e-learning is at the postsecondary level with 
many fewer studies at the primary and secondary levels and fewer still in other theme areas. 

We also examined the primary literature more closely by considering the types of evidence 
reported (see Table 5). The majority of the evidence was generated by qualitative studies which 
are especially well suited to descriptive investigations, and which may result in the generation of 
useful hypotheses. 

A related picture emerged when we examined the types of research designs employed in the 
primary research. See Table 6. Few of the studies used experimental or quasi-experimental designs 
























to explore causal explanations about the nature of what works. 

In the primary research, the type of evidence and the type of research design was not distributed 
with equal proportion across the theme areas. For example, the percentage of RCTs was highest in 
the health and e-learning theme area. 


I able 4. Cross-la bn la lion of evidence by CCI. Theme Area 




CCL T hem e Areas 


Dik'u in cnt 
Source 


Adult 

Education 

Early 

Childhood 

Education 

Elementary/ 

Secondary 

Education 

Post- 

secondary 

Education 

Health 

and 

Eeamimt 

>01 

Specified 

Total 

General 

Count 

4 

7 

94 

114 

5 

11 

Ul 

V* 

public 

opinion 

% 

14 30% 

87 50*/. 

4000% 

3200% 

810% 

29.70% 

32 40% 

Trade/ 

Count 

10 

0 

32 

70 

10 

9 

131 

practitioners' 

position 

*/. 

35.70% 


13.60% 

19.70% 

16.10% 

24.30% 

18 00% 

Policy 

Count 

3 

1 

24 

43 

16 

1 

88 

making 

reports 

% 

10.70% 

12 50% 

10.20% 

1210% 

25.80% 

270% 

12.10% 

Scholarly 

Count 

3 

0 

34 

51 

21 

11 

120 

academic 

reviews 

•/. 

10.70% 


14 50% 

14 30% 

33.90% 

29.70% 

16 50% 

Primary 

Count 

8 

0 

51 

78 

10 

5 

152 

empirical 

research 

% 

28.60% 


21.70% 

21.90% 

16.10% 

13.50% 

20 90% 

1 otal 

Count 

28 

8 

235 

356 

62 

37 

726 


% 

100.00% 

100.00% 

100.00% 

100.00% 

100.00% 

100.00% 

100,00% 


Table 5. Analysis of the primary research by t\pe of evidence 


Type of Evidence 

Frequency 

Percent 

Survey Data 

26 

17.1 

Quantitative 

41 

27 

Qualitative 

85 

55.9 

Total 

152 

100 


Table 6. Analysis of the primary research by t\pe of research design 


Research Design 

Frequency 

Percent 

Pre-experimental 

77 

50.7 

Correlational 

44 

28.9 

Quasi-Experimental 

14 

92 

RCT 

10 

6.6 

Unclassified 

7 

4.6 

Total 

152 

100 


Analysis of predictors and outcomes: All sources of evidence 





































We took a subset of the variables used to describe the literature in the Argument Catalogue 
codebook and used these as predictor and outcome variables. The predictor variables included 
pedagogical features (such as the provision of professional training), learner characteristics (such 
as gender), context effects (such as school location), and technology factors (such as intervention 
type). We recoded all of these predictor variables, except for the technology factors, to have 
dichotomous values, where "1" meant that the factor was considered in the document and "0" 
meant that the factor received no attention. The advantage of this recoding is that it allowed us to 
include sources of evidence where these features were mentioned as influential but no directional 
relationship was specified. For example, a newspaper article describing female students' use of 
technology would be an instance of where we would code for gender as a relevant issue, (even 
though there might not be evidence in the article speaking directly to the benefit or gain of e- 
learning). Similarly, a policy document describing the need to analyse the effects of gender on the 
digital divide would also be coded as "1" for the gender factor. 

Technology factors were extracted from the open-ended coding and were treated as categorical 
predictors. The means and standard deviations for the dichotomous predictors are given in the 
upper portion of Table 7. The means are equivalent to the proportion of articles that included the 
predictor. For example, the mean for professional training is 0.116 that means that professional 
training was specifically mentioned as important for effective e-learning in 11.16% of the 
documents. 

Many of the articles raised issues about the importance of pedagogy in the effective application of 
e-learning. Of those, few addressed the issue of professional training while almost half mentioned 
infrastructure support and logistics, and slightly more than one-third were relevant to course 
development. 

Few studies mentioned learner characteristics and school context effects as relevant to e-learning. 
This was somewhat of a surprise as we expected aspects of the digital divide, including gender, 
race/ethnicity, learners with special needs, and rural/urban location to feature more prominently in 
the literature. It is worth noting, however, that the prevalence of a predictor among the collection 
of documents does not necessarily relate to the extent to which a variable explains impacts of e- 
learning. 

The last collection of predictors examines several aspects of technology. We used predictors which 
were recoded from the open-ended emergent factors identified by the coders and summarized 
initially in short text remarks. The resulting three variables were: Context of technology use (i.e., 
distance education, in class, blended, unspecified); Type of tools used (i.e., internet/intranet/on- 
line/web, virtual reality/learning objects/simulations, technology integration—computers and 
software for particular purposes, unspecified); and Intervention type (i.e., instructional, 
communicative, organizational, analytical/programming, recreational, expansive, creative, 
expressive, evaluative, informative, and unspecified/missing). Most of the documents focused on 
distance education and hybrid uses of technology; about one-third focused on the use of 
technology in face-to-face learning situations. Half of the tools used were network-based 
technologies but almost 20% looked at aspects of technology integration. Finally, the functional 
uses of technology described in the documents ranged from instructional to communicative and 
creative. Otherwise, we noted that the percentage of documents which described aspects of 
technology use overall was higher than the percentage of documents that described aspects of 
pedagogy. 



Table 7. Analysis of predictor variables 


Pedagogy - Professional Training 

Mean =0.11 

SD = 031 

Pedagogy - Course Development 

Mean =0.38 

SD = 0.48 

Pedagogy - Infrastructure Logistics 

Mean =0.47 

SD = 0.49 

Type of Learners - Special Needs 

Mean =0.04 

n 

o 

II 

8 

Type of Learners - General Population 

Mean =026 

SD =0.44 

Type of Learners - Gifted 

Mean =0.01 

SD =0.083 

Type of Learners - Gender Issues 

Mean =0.02 

SD = 0.15 

Type of Learners - Ethnic Race Aboriginal Religion 

Mean =0.01 

SD = 0.13 

C ontext - Lo cati on 


Mean =0.05 

SD = 023 

Context - School Setting 

Mean =0.05 

SD = 022 

Technologv recoded as 

DE.F2F Hybrid 

Frequency 

Percent 

DE 

223 

30.7 

Face-to-face 

133 

18.3 

Hybrid 

91 

12.5 

DNA Unspecified 

279 

38.4 

Technology' recoded as the type of 
tools, approaches 

Frequency 

Percent 

All network-based technologies: 
Internet Web-based CMC, etc. 

367 

50.6 

Virtual reality, simulations, etc. 

23 

32 

Technology' integration 

143 

19.7 

DNA Unspecified 

193 

26.6 

Technology- recoded by- major function 

Frequency 

Percent 


































Instructional presentational 

98 

13.5 

Communication 

111 

15.3 

Organizational 

9 

12 

Analytical 

6 

0.8 

Expansive 

2 

0.3 

Creative 

24 

3.3 

Expressive 

7 

1 

Evaluative 

4 

0.6 

Informative 

7 

1 

DNA Unspecified 

458 

63.1 


Table 8 presents an analysis of the impact measures by source of evidence. Some of the impact 
measures were derived from actual measured results, but most were coding judgments and should 
be interpreted as perceived impacts. 

The impact measures were recoded so that +1 was a positive outcome, 0 was a neutral or mixed 
outcome, and -1 was a negative outcome. Of the 726 documents we coded, the most represented 
impact was Learning flexibility with 324 documents (43%), followed by Achievement with 250 
documents (33%), Interactivity/Communication with 237 documents (31%), Meeting Social 
Demands with 225 documents (30%), Motivation Satisfaction with 204 documents (27%), Cost with 
100 documents (13%), and Attrition/Retention with 44 documents (5%). Of the 100 documents that 
mentioned cost issues, only six were from primary research evidence. Only 250 or about one-third 
specifically mentioned achievement impacts of e-learning. Oneway between groups analysis of 
variance were conducted on each outcome by source of evidence (IV). There was a significant 
difference on achievement impacts only (p < .05) although all average effects were positive. 
Practitioner positions were most positive while general/public opinions were least positive. Total 
impacts are shown graphically in Figure 3. 
















Figure 3. Perceived impacts of e-learning 


lable 8. Analysis of e-learning impacts hy source of evidence 


Document Source 

Achievement 

Motis at ion/ 
Satisfaction 

Interactivity/ 
Comm unication 

Meeting 

Social 

Demands 

Attrition/ 

Retention 

Learning 

flexibility 

Cost 

General public 

Mean 

049 

0 85 

0 74 

0 86 

0 12 

1 00 

030 

N 

55 

60 

54 

96 

16 

122 

39 


SD 

074 

0 48 

0.61 

047 

095 

0 

0.79 

Trade.' 

Mean 

089 

073 

085 

080 

033 

0.92 

0.125 

practitioners’ 

N 

29 

26 

47 

36 

3 

68 

16 

position 

SD 

0.30 

0.60 

0.41 

0.40 

1.15 

0.26 

0.80 

Policy making 
reports 

Mean 

N 

0 68 

0 50 

082 

0 86 

0 42 

091 

025 

35 

18 

28 

45 


46 

28 

SD 

047 

0 51 

0.47 

0.34 

0.53 

0.28 

0.58 

Scholarly 

Mean 

0.53 

0.65 

0.83 

0.96 

0.16 

0.84 

-0.18 

academic 

N 

56 

43 

49 

29 

12 

52 

11 

reviews 

SD 

060 

048 

0.37 

0 18 

0.71 

0.36 

0.60 

Prim an 

Mean 

065 

064 

0.76 

0.84 

0.83 

0.88 

-0.16 

empirical 

N 

75 

57 

59 

19 

6 

36 

6 

research 

SD 

050 

058 

046 

050 

040 

0.31 

0.75 


Mean 

0.62 

0.70 

0.79 

0.86 

0.29 

0.93 

0.18 

Total 

A r 

250 

204 

237 

225 

44 

324 

100 


SD 

0.57 

0.53 

0.47 

0.41 

0.79 

0.24 

0.73 


Table 9 presents an analysis of the impact measures by theme area. All the average impacts are in 
a positive direction. But note the dearth of evidence on learning and motivation for certain theme 
areas, particularly adult education, early childhood learning, and health and e-learning. 




























Table 9. Analysis of e-learning inywcis by CCI. Thane Areas 


CCI. IliemeAreas 

\ chi element 

Moti\ at ion/ 
Satisfaction 

Interactivity/ 

C omm uniciition 

Meeting Social 
Demands 

Attrition/ 

Retention 

1.entiling 
flexibility 

Coat 


Mean 

090 

0.91 

070 

091 

too 

0.92 

0.00 

Adult 









Education 

N 

10 

11 

10 

II 

1 

13 

1 


SD 

032 

0 30 

048 

030 


028 



Mean 

0 17 

000 

000 

075 




Early 









Childhood 

N 

6 

3 

1 

4 

0 

0 

0 


SD 

098 

1.00 


050 





Mean 

0 57 

080 

0 78 

090 

0 22 

093 

0,03 

Elementary/ 









Secondary 

N 

95 

64 

74 

61 

18 

91 

30 


SD 

063 

044 

0 53 

040 

0 88 

025 

076 


Mean 

0.67 

0.65 

0.77 

0.83 

0.33 

0.92 

0.21 

Post secondary 









Education 

A r 

103 

107 

117 

121 

24 

182 

56 


SD 

0.53 

058 

050 

0.45 

076 

0.27 

0.70 


Mean 

0 70 

0 79 

1 00 

0 91 

000 

1 00 

050 

Health and 









Learning 

JV 

23 

14 

28 


1 

29 

12 


SD 

0.47 

0.43 

000 

0.29 


0.00 

0.80 


Mean 

054 

040 

086 

1.00 


1.00 

1.00 

Not Specified 

N 

13 

5 

7 

6 

0 

9 

1 


SD 

0 52 

055 

© 

000 


000 



Mean 

0.62 

0.71 

0.80 

0.87 

too 

0.94 

0.18 

Total 

N 

250 

204 

237 

225 

44 

324 

too 


SI) 

0.58 

0.54 

0.48 

0.41 

0.79 

105 

0.73 


We next examined the interrelationship among the impact measures. See Table 10. As the 
measures are strongly correlated (except for "Cost") we formed a composite measure, "Impact," 
which is the average of the individual measures (except "Cost," and excluding missing values). The 
composite measure is highly correlated with each of the individual measures and has the advantage 
of representing a larger number of documents than any single measure alone. It should also be 
noted that measures of impact include both measured and perceived (speculative or estimated) 
assessments of the effects of e-learning. 











Table 10. Intercorrelations among individual and composite impact measures 


Impact 

Measures 

Motivation/ 

Satisfaction 

Interactivity/ 

Communication 

Meeting 

Social 

Demands 

Attrition' 

Retention 

Learning 

flexibility 

Cost 

Achievement 

0.65* 

0.61* 

0.55* 

0.69* 

0.53* 

0.21 

N 

116 

100 

79 

20 

109 

30 

Motivation 

Satisfaction 


0.67 

0.66 

0.09 

0.42* 

-0.36 

N 


111 

75 

23 

116 

34 

Interactivity 

Communicatior 



0.73* 

-0.09 

0.4S* 

0.015 

N 



91 

22 

161 

3S 

Meeting Social 
Demands 




a 

0.43* 

0.14 

N 





14S 

49 

Attrition 

Retention 





-0.07 

0.43* 

N 





29 

17 

Learning 

flexibility 






022 

N 






6S 

Impact of 
e-leaming 
(composite 
mean) 

0.93* 

O.SS* 

0.S7* 

0.S9* 

0.733* 


N 

250 

204 

237 

225 

44 



*p <0.05; a -No correlation couldbe computed. 


Which Predictor Variables are Associated with Perceived Technology 
Impacts Overall? 

In this section we explore the relationship between the individual predictor variables and the 
composite measure of impact for all the sources of evidence taken together. Doing so combines 
evidence and opinion without regard for the direction of the relationship between variables coded 
dichotomously (e.g., applies, does not apply). In other words, we can conclude whether a factor is 
associated with technology impacts overall but not whether the direction of the influence is positive 
or negative. 

In Table 11 we report the overall perceived impacts of e-learning across sources of evidence. The 
composite Impact measure is positive in a majority of the documents regardless of the source of 
evidence. If all the documents from a single source were positive the mean would be +1.00 and the 
standard deviation would be zero, indicating uniform evidence. That most of the averages were 
high, with small standard deviations, suggests that the view that e-learning has deleterious effects 
is an unpopular one, which has received little general support in the primary evidence and literature 
reviews of evidence. 
















Table 11. Overall perceived impacts of e-learning across sources of evidence 


Document Source 

Mean 

N 

SD 

General public opinion 

0.77 

190 

0.52 

Trade practitioners’ position 

0.S2 

90 

0.40 

Policy making reports 

0.76 

67 

0.39 

Scholarly academic reviews 

0.6S 

95 

0.49 

Primary empirical research 

0.67 

128 

0.46 

Total 

0.74 

570 

0.47 

F(4, 565) = 1.95,;? =0.10 





An examination of the perceived impacts of e-learning across theme areas showed uniformly 
positive results except for early childhood education. See Table 12. There were only seven studies 
in this area. This may indicate apprehension about using e-learning for children prematurely, the 
difficulty of research, or a general lack of interest in the topic for young children. 

According to Table 13, there were significant differences in the overall perceived impacts of e- 
learning depending on the context of technology use. Impacts are larger in distance learning 
contexts and smaller in face-to-face learning situations. The benefits of e-learning at a distance 
may be enhanced when technology is understood to be a necessary condition for learning and 
instruction. 

Table 14 summarizes the significant differences in perceived impact when the nature of the 
technology tools are examined separately. Documents which highlight the use of virtual reality and 
simulations have the highest perceived impact of e-learning, though the total number of documents 
containing them is rather limited. 


Table 12. Overall perceived intpacts of e-learning across theme areas 


CCL Theme Areas 

Mean 

N 

SD 

Adult Education 

0.79 

23 

0.38 

Early Childhood Education 

0.17 

7 

0.S9 

Elementary Secondary Education 

0.72 

ISO 

0.51 

Postsecondary Education 

0.75 

288 

0.45 

Health and Learning 

0.80 

51 

0.40 

Not Specified 

0.73 

21 

0.39 

Total 

0.74 

570 

0.47 


F(5, 564) =2.302,/) <0.05 

























Table 13. Overall perceived impacts of e-learning across the contexts of 
technology use 


Context of Technology Use 

Mean 

.V 

SD 

Face-to-Face 

0.60 

108 

0.58 

Hybrid 

0.75 

80 

0.46 

DE 

0.79 

183 

0.41 

DNA Unspecified 

0.76 

199 

0.45 

Total 

0.74 

570 

0.47 

.F(3, 566) =4.10,/? <0.01 





Table 14. Overall perceived impacts of e-learning across technology• took 


Technology Tools 

Mean 

N 

SD 

Technology integration 

0.65 

117 

0.53 

All network-based technologies: 
Internet Web-based CMC, etc. 

0.72 

138 

0.49 

Virtual reality, simulations, etc. 

0.81 

16 

0.40 

DNA Un spe cifie d 

0.78 

299 

0.44 

Total 

0.74 

570 

0.47 

F(3. 566) =2 30,p =0.084 





Table 15 focuses not on how the technology tools were designed but how the technology was used 
pedagogically. Perceived impacts of technology are believed to be larger for communication and 
student-led purposes and smaller for instructional and informative purposes. A possible 
interpretation of this finding focuses on using technology as an interactive tool for learning versus 
using technology for information gathering and information transmission. 


Table 15. Overall perceived impacts of e-learning across pedagogical uses of 
technology 


Pedagogical Uses of Technology 

Mean 

N 

SD 

Instructional Informative 

0.63 

82 

0.46 

Communication 

0.77 

344 

0.46 

Student Applications 

0.77 

47 

0.46 

DNA Un spe cifie d 

0.69 

97 

0.50 

Total 

0.74 

570 

0.47 


F(3, 566) =2A2,p =0.065 


Table 16 shows that the perceived impact of e-learning is higher among the documents that 
specifically mention the inclusion of professional training in the uses of technology for learning. This 
is consistent with the notion that professional development is one key to the effectiveness of 
educational technology. 



































Table 16. Overall perceived impacts of e-learning for professional training 


Pedagogy - Pl ot Training 

Mean 

N 

SD 

Applies 

0.86 

65 

031 

Does not apply 

0.72 

505 

0.48 

Total 

0.74 

570 

0.47 

F(l, 568) =4.69,<.05 





Table 17 shows that the perceived impacts of e-learning are higher in documents that mention the 
importance of addressing technology infrastructure and logistics. 


Table 17. Overall perceived impacts of e-learning for infrastructure / logistics 


Pedagogy - Infrastructure Logistics 

Mean 

N 

SD 

Applies 

0.81 

265 

0.40 

Does not apply 

0.6S 

305 

0.51 

Total 

0.74 

570 

0.47 

F(l, 568)= 10.43,;? = 0.001 





Tables 18 and 19 examine two aspects of context—location and setting. Both factors are related to 
perceived impacts. 


Table 18. Overall percehed impacts of e-learning for location 


Context-Location 

Mean 

N 

SD 

Applies 

0.89 

42 

034 

Does not apply 

0.73 

528 

0.48 

Total 

0.74 

570 

0.47 

F (1, 568) = 4.77, p < .05 





Table 19. Overall perceived impacts of e-learning for school setting 


Context- School Setting 

Mean 

N 

SD 

Applies 

0.88 

36 

022 

Does not apply 

0.73 

534 

0.48 

Total 

0.74 

570 

0.47 

F(l, 568) = 3.51,p =0.061 





Analysis of Predictors and Outcomes: Research Reviews and Primary Canadian Research 

These two literatures together include 223 documents. So that we could examine best practices 
and procedures in the two literatures combined, we first tested the difference between them in 
terms of our composite impact measure using one-way ANOVA. Results revealed that the two 
bodies of literature were not significantly different regarding the impact of e-learning and so their 
data could be treated together. Means and standard deviation, plus the F-ratio, are shown in Table 
20 . 














Table 20. Perceived impacts of e-learning: Comparison of sources of research 
evidence 


Source of Document 

N 

Mean 

SD 

Scholarly academic reviews 

95 

0.68 

0.49 

Primary empirical research 

128 

0.67 

0.46 

Total 

223 

0.67 

0.47 

F(l,221) = .020,/? >0.05 





Next, we were interested in knowing if the results of the foregoing analyses could be generalized 
across all of the CCL Theme Areas. That is, are there differences in impact that differentiate adult 
education students, for instance, from those involved in Health Learning? Table 21 shows the 
results of this analysis. We found that mean impact did not vary across categories of Theme Area. 
The only caveat here is the low cell frequency for Adult Education and Health and Learning. Note 
that the category Early Childhood Education does not appear in the table. This is because we found 
that there are no documents that relate to this theme area in the literature we reviewed. 


Table 21 .Perceived impacts across CCL theme areas 


CCL Theme Areas 

N 

Mean 

SD 

Adult Education 

11 

0.68 

0.46 

Elementary Secondary Education 

68 

0.71 

0.48 

Postsecondary Education 

109 

0.65 

0.49 

Health and Learning 

25 

0.68 

0.47 

Not Specified 

10 

0.65 

03S 

Total 

223 

0.67 

0.47 


F(4,21S) = .133,/? >0.05 


We began to address the issue of best practices by examining the context within which technology 
is applied in terms of its global impact. Three context categories were coded, ranging from standard 
face-to-face instruction to distance education (DE) where technology is a requirement of 
implementation. We found (Table 22) that these categories of context did not differ. 


Table 22. Perceived impacts: Comparison among categories of “context of 
technology use” 


Context of Use 

iV 

Mean 

SD 

Face-to-Face 

51 

0.61 

0.48 

Hybrid 

40 

0.73 

0.40 

DE 

55 

0.64 

0.48 

DNA Unspecified 

77 

0.70 

0.50 

Total 

223 

0.67 

0.47 

F(3,219) = .672,/? >0.05 





We examined the impact of various technologies (see Table 23) and found that, in terms of impact, 
none was better than another. However, note that the largest category frequency is 


























DNA/Unspecified, suggesting that the research literature, unfortunately, does not always make the 
kind of technology investigated explicit. 


Table 23. Perceived impacts: Comparison among categories of “technology 
type” 


Technology Type 

JV 

Mean 

SD 

Technology integration 

51 

0.58 

0.53 

All network-based technologies: 
Internet Web-based CMC, etc. 

62 

0.63 

0.48 

Virtual reality, simulations, etc. 

9 

0.77 

0.44 

DNA Unspecified 

101 

0.70 

0.44 

Total 

223 

0.67 

0.47 


F(3,219) = .941,/> >0.05 


One of the most important issues relating to best practices in e-learning is how technology is 
applied to instructional settings. The variable that addresses this issue is shown in Table 24 and is 
referred to as Pedagogical Implementation of Technology. The levels shown in Table 24 range from 
teacher-centered uses of technology to student-centered uses of technology. While the A-ratio does 
not reach significance, it is close enough to interpret the post hoc analysis that came after ANOVA. 
Post hoc analysis indicated that Instructional/Informative applications of technology significantly 
differed from Student Applications. 


Table 24. Perceived impacts: Comparison among categories of “pedagogical 
implementation of technology” 


Pedagogical Implementation 

N 

Mean 

SD 

Instructional Informative 

33 

0.49 

0.49 

Communication 

106 

0.72 

0.49 

Student Applications 

29 

0.77 

0.41 

DNA Un spe cifie d 

55 

0.65 

0.45 

Total 

223 

0.67 

0.47 

F(3,219) =231,p = .072 





Among the variables of interest to both practitioners and policy makers is whether the introduction 
of professional training has an impact on the outcomes of technology use. Does it matter if teacher 
training, workshops, online help sites, etc. are worth the effort and expense associated with them? 
The research literature suggests that it does matter. Documents which contained evidence of the 
implementation of professional development had a higher impact mean than documents that did not 
mention it (See Table 25). 


Table 25. Perceived impact of Professional Training 


Professional Training 

N 

Mean 

SD 

Applies 

36 

0.86 

032 

Does not apply 

187 

0.64 

0.49 

f (df = 70.49) =2.68,/? < .01 

























Similarly, we considered whether course development has an impact on the successful 
implementation of e-learning. While Table 26 indicates that the mean for global impact is higher in 
documents that consider course development, there is no significant difference. 


Table 26. Perceived impact of Course Development 


Course Development 

N 

Mean 

SD 

Applies 

117 

0.71 

0.48 

Does not apply 

106 

0.63 

0.47 

t (df = 221) = 1 24, p > .05 





A similar finding resulted (See Table 27) when the impact of Infrastructure and Logistics was 
considered. 


Table 27. Perceived impact of Infrastructure/Logistics 


Infrastructure Logistics 

.V 

Mean 

SD 

Applies 

94 

0.71 

0.45 

Does not apply 

129 

0.64 

0.49 

r = (df= 208.34)= 1.11,/? > .05 





The results of these analyses indicates an overall relationship between the predictor variables, 
namely the nature of technology use and pedagogical factors, and general impacts of e-learning. 
Furthermore, these relationships are generally similar to our conclusions when we included all 
sources of evidence in the analyses. 

However, this should not be taken to mean that in given circumstances, with particular learner 
groups that these conditions apply universally, for at least two reasons. First, there is variability 
associated with the predictors and outcomes; individual relationships appear only modest in size. 
Second, and especially important, the quality and quantity of the evidence does not allow us to 
conclude with certainty the factors which impact on e-learning. 

Quantitative Summary of the Canadian Primary Research on E-learning 

What is a quantitative summary? As a final stage in this research, we undertook to identify the 
primary e-learning studies from the Canadian context that could be summarized quantitatively. This 
summary resembles a meta-analysis in its methodological approach, but is not as inclusive as a 
meta-analysis might be expected to be. The methodological approach to meta-analysis involves 
synthesizing the experimental and quasi-experimental studies from which an effect size can be 
extracted. An effect size is the standardized mean difference between a treatment condition (e- 
learning) and control condition (no e-learning). An effect size, therefore, is interpreted in units of 
standard deviation, and can be negative or positive. A positive effect size says that the treatment 
group outperformed the control group. A negative effect size says the reverse. Generally speaking, 
effect sizes up to ±0.25 are considered small and effect sizes up to ±0.75 are considered moderate. 
Effect sizes over ±0.75 are large, and when positive suggest that the treatment had a noticeable 
effect compared to the control condition. 

A distribution of effect sizes can be averaged to produce an overall summary. The same standards 
of interpretation that apply to individual effect sizes also apply to the average. However, the 
variability among effect sizes around the mean must also be considered. Widely dispersed effect 






sizes (heterogeneous) are not representative of a single parameter in the population of effect sizes. 
In other words, the mean effect size cannot be unambiguously distinguished from other mean effect 
sizes in the same population. 

When an overall mean effect size is heterogeneous, it is usually appropriate to explore this 
variability by introducing moderator variables. We did so here by exploring the relationship between 
effect size and type of research design. 

Method of selecting studies. Our goal was to identify the studies, from the Canadian primary 
research literature, that could be included in a meta-analysis, to attempt to describe the measured 
impact of e-learning based on high-quality studies. We examined the 152 studies that we have 
previously described and analyzed and only found a total of 7 that were truly experimental (i.e., 
random assignment) and 10 that were quasi-experimental (i.e., not randomized but possessing a 
pretest and a posttest). The presence of a pretest makes a quasi-experimental design more 
interpretable than a non-randomized posttest only design. Within these 17 studies we identified 29 
independent effect sizes (i.e., studies often contain more than one finding). Two raters, working 
independently, judged the number of effect sizes that could be derived from each study (89% 
agreement) and the calculation of the effect size that was extracted in each case (97% 
agreement). 

Analysis and results. It is generally known that small sample means suffer greatly from the 
influence of outlying findings and so we decided to consider the composite collection of effect sizes 
as "the composite impact of e-learning," rather than analyzing them separately by type of outcome 
measure. For the 29 effect sizes, Hedges' g+ (unbiased mean) was +0.117, by the standards 
previously described, a small effect (see the last line in Table 28). Accordingly, approximately 54% 
of the e-learning participants performed at or above the mean of the control participants (50 th 
percentile), an advantage of 4%. However, the heterogeneity analysis was significant (the Q-value 
in Table 28), indicating that the effect sizes were widely dispersed. It is clearly not the case that e- 
learning is always the superior condition for educational impact. Our search for the ideal conditions 
continues. Elsewhere in this review we have noted the possible importance of pedagogical features 
as they relate to the integration of technology into learning. We continue to consider this to be an 
important direction for future research. 

We then undertook an examination of the differential effects of true experiments and quasi¬ 
experiments. This analysis is also shown in Table 28. The mean effect size of the 20 true 
experiment effects was +0.109 and the mean effect size of the 9 quasi-experimental effects was 
+0.245. 


Table 28. E-learning effect sizes broken down by research design. 


Design Types 

Studies 

Effect 

Sizes 

95% confidence 

interval 

Heterogeneity 

k 

g + 

Lower 

limit 

Upper 

limit 

O-value 

dm 

T^- 

value 

True Experimental 
Quasi- 

Experimental 

20 

0.108 

0.062 

0.155 

79.068 

19 

0.000 

9 

0245 

0.066 

0.423 

14.570 

S 

0.068 

Composite 

29 

0.117 

0.072 

0.162 

95.746 

2S 

0.000 














Discussion 

Introduction 

The proliferation of, and interest in e-learning in Canada is unquestionable. However, it is somewhat 
uncertain how e-learning will proceed in the future, given the magnitude of the investment required 
to develop it, sustain it and cause it to grow. We undertook this review to provide answers to a 
host of questions regarding research and development of e-learning over the past five years. While 
history is not always the best guide to future decision-making, it does provide a level of evidence 
that goes beyond decision-making based on opinion, politics, conjecture and posturing. We believe 
that the foregoing review will provide a set of lenses through which those areas of successful and 
unsuccessful implementation can be brought into focus. 

An uncritical assessment of the evidence from myriad sources provides a fairly consistent and 
positive picture. Conclusions from Canadian primary research, international literature reviews, 
policy documents, media reports, and practitioner publications are mostly favourable. Yet a closer 
examination of the evidence paints a less convincing portrait or "rough sketch." In Canada there is 
a lack of evidence in some theme areas, notably early childhood learning, and a lack of 
experimental and quasi-experimental evidence that would allow unambiguous causal conclusions to 
be drawn about effectiveness. The quality and scope of the research evidence does not match the 
time, cost and resources that have been and will be dedicated to the development and 
implementation of e-learning. One of our most striking conclusions—one that bears close 
consideration by practitioners and policy makers alike—is the need for programs of development for 
new initiatives that have high-quality research and evaluation programs or components built-in as a 
forethought, not an afterthought. And we believe there is a need for large-scale, longitudinal and 
intensive implementations of e-learning which go beyond investigations of short-term impact. Such 
initiatives require the attention of specially trained educational technologists, working alongside 
content experts and other educators. 

In the health sciences, new methods are implemented following a progression from small scale, 
controlled laboratory experimentation, to controlled field trials, to wide scale field investigations 
and follow-up. In Canada's e-learning endeavours, we need to consider whether such a model of 
inquiry may bear fruit and whether the expenses of pan-Canadian efforts at scalable inquiry are 
truly worth the costs. 

What We Know: Consensual Validation 

We were surprised at the remarkable consistency that emerged across the sources of evidence 
and, to a lesser extent, across the CCL theme areas. Opinions among policy makers, practitioners, 
the general public, plus evidence from Canadian primary research and a variety of literature 
reviews, all point to the positive impacts of e-learning on achievement, motivation, communication, 
learning flexibility and meeting social demands. But the consensual evidence alone does not reveal 
what accounts for these positive impressions across the multiple constituencies represented in this 
study. 

What We Know: The Evidence 

Major Messages: Sample of Document Conclusions 

In the Results, we included a section on the analysis of the most representative messages found in 
each of the five document sources: general public opinion; practitioner literature; policy 
documents; scholarly reviews; and primary Canadian research. What follows is a summary of the 
major messages that were derived from each of these sources. 



General Public Opinion: ,. , , . 

• E-learning is aTapidly growing field in education. 

• E-learning provides greater access to educational programs. 

• Funding the high costs of e-learning may divert resources away from other educational priorities. 

• There is fear of a negative impact on the development of children's creative skills. 

• Teachers and classrooms will remain essential in the world of e-learning. 


Practiti o n ers: 

• E-learning increases accessibility, learning flexibility, and opportunities for learning. 

• E-learning requires careful attention to instructional design, pedagogical planning, professional training 
and fiscal support. 

• We need new policies and strategies to meet the emerging social demands of educational technology. 

|fcy*ma^rs*are mostly favourable towards e-learning. 

• There is a need to bridge the gap between theory, research, and practice. 

• Technology should be introduced and used only in appropriate contexts. 

• There are four major reasons for using e-learning: economic competitiveness; educational attainment, 
increased access, and catalyst for educational change. 

Reviews: 

• The reviews of e-learning range from neutral to positive; it is at least as effective as traditional 
instruction. 

• We need to address design issues and new strategies for teaching and learning. 

• Effective e-learning requires the presence of immediate, extensive, and sustained support. 

• There is an absence of strong empirical evidence to support the use of e-learning. 

^ >f "» fT ^) , rKe ,S |^a < rnem are better prepared to use e-learning effectively than others. 

• Effective instructional design fore-learning does not resemble traditional pedagogical methods. 

• Teachers require professional development and training to use technology effectively. 

• Collaborative methods afforded by online technologies facilitate the development of higher-order thinking. 

• E-learning provides disabled students with previously unavailable educational opportunities. 


Findings from Analyses of Data 


We examined the data from a number of perspectives, including that of its quality as evidence. 
Because of the categories of sources we chose to examine, from public opinion to primary research, 
the quality of evidence varies accordingly. In the primary research literature we examined the kinds 
of research designs that were used. We found that over half of the studies conducted in Canada are 
qualitative in nature, with the other nearly half split between surveys and quantitative studies 
(correlational and experimental). When we looked at the nature of the research designs, again, 
51% are qualitative case studies and 15.8% are experimental or quasi-experimental studies. It 
seems that studies that can help us understand "what works" in e-learning settings are 
underrepresented in the Canadian research literature. 


Generally speaking, the data can be classified as either outcomes (we refer to them as "impacts.") 
or predictors. Outcomes/impacts are the perceived or measured benefits of e-learning, whereas 
predictors are the conditions or features of e-learning that can potentially affect the 
outcomes/impacts. The impacts were coded on a positive to negative scale and included: 1) 
achievement; 2) motivation/satisfaction; 3) interactivity/communication; 4) meeting social 
demands; 5) retention/attrition; 6) learning flexibility; and 7) cost. Based on an analysis of the 
correlations among these impacts, we subsequently collapsed them (all but cost) into a single 
impact scale, ranging from -1 to +1. We found, generally, that the perception of impact or actual 
measured impact varies across the types of documents. They appear to be higher in general 
opinion documents, practitioner documents and policy making reports than in scholarly reviews and 
primary research. While this represents an expression of hope for positive impact, on the one hand, 
it possibly represents reality on the other. Where there were sufficient documents to examine and 
code, impact was high across each of the CCL Theme Areas. Health and Learning was the highest, 



with a mean of 0.80 and Elementary/Secondary was the lowest, with a mean of 0.77. However, 
there was no significant difference between these means. 

The perceived impact of e-learning and technology use was higher in distance education, where its 
presence is required (Mean = 0.80) and lower in face-to-face instructional settings (Mean = 0.60) 
where its presence is not required. Network-based technologies (e.g., Internet, Web-based, CMC) 
produced a higher impact score (Mean = 0.72) than straight technology integration in educational 
settings (Mean = 0.66), although this difference was considered negligible. Interestingly, among the 
Pedagogical Uses of Technology, student applications (i.e., students using technology) and 
communication applications (both Mean = 0.78) had a higher impact score than instructional or 
informative uses (Mean = 0.63). This result suggests that the student manipulation of technology in 
achieving the goals of education is preferable to teacher manipulation of technology. 

In terms of predictor variables (professional training, course design, infrastructure/ logistics, type 
of learners—general population, special needs, gifted, gender issues and 
ethnicity/race/religion/aboriginal—location, location, school setting, context of technology use, type 
of tool used and pedagogical function of technology) we found the following: 

• Professional development was underrepresented compared to issues of course design and 
infrastructure/logistics. 

• Most attention is devoted to general population students, with little representation of special needs, the 
gifted students, issues of gender or ethnic/race/religious/aboriginal status. 

• The greatest attention is paid to technology use in distance education and the least attention is paid to 
the newly emerging area of hybrid/blended learning. 

• The most attention is paid to networked technologies such as the Internet, the WWW and CMC and the 
least paid to virtual reality and simulations. 

• Using technology for instruction and communication are the two highest categories of pedagogical use. 

Our quantitative summary of the primary Canadian studies for which effect sizes could be 
calculated tends to support the general impression of the positive benefits of e-learning. While the 
mean effect size is not large ( g+ = 0.117), it is nonetheless in a positive direction. However, the 
wide variability suggests that the effectiveness of e-learning is not always guaranteed. Given that 
we as educators are still on the threshold of understanding how to design courses to maximize the 
potentials of technology, this finding represents a hopeful sign. More attention must be given to 
exploring the interactive effects of learning about technology and learning with technology, while 
realizing all the while that it is not the panacea that some suggest it to be. 

The CCL Objectives 

The objectives of this review, as stated by the Canadian Council on Learning, are as follows: 1) 
Identify and verify through research the most effective practices and procedures to promote 
learning; 2) Identify major gaps in our knowledge and understanding of e-learning; and 3) Identify 
the most promising lines of inquiry for addressing those gaps. 

The approach that we took in this review—the use of frequency counts, weak measures of 
outcomes and relatively few coded predictor variables—does not readily present us with evidence 
of best practices and "what works" in e-learning. This is the job that a meta-analysis could 
accomplish by estimating the average strength of various instructional treatments and the 
variability associated with them. Even if we had originally intended to do this on the primary 
empirical literature of Canada, a task that could have been managed in the time frame of this 
review, the relatively small corpus of quantitative research conducted in Canada would have 
rendered this task nearly impossible to achieve, and would probably have resulted in our having to 



say "we can't tell from the data." However, this approach to systematic review has revealed where 
many research efforts have been placed and where the gaps in our knowledge lie. Overall, we know 
that research in e-learning has not been a Canadian priority; the culture of educational technology 
research, as distinct from development, has not taken on great import like it has in the U.S., except 
as it appears to have happened in the business sector. Historically, there have been few successful 
centres, institutes or networks of research in educational technology that would generate the kinds 
of Canadian research and evaluation evidence of learning impacts upon which strong conclusions 
can be based. Likewise, there is no mandate for evidence-based policy making (as there is in the 
U.K., for instance) that places issues such as research in the development of e-learning as a major 
pan-Canadian research priority. 

In addition, there appears to have been a disproportionate emphasis on qualitative research in the 
Canadian e-learning research culture. While we laud any and all quality research efforts, the 
relative de-emphasis on quantitative research means that its areas of strength—those types of 
questions that can be answered through experimental and quasi-experimental studies—are not as 
well represented in the literature as would be desirable. Does this mean that our conclusions for 
innovation must be based primarily on investigation in foreign contexts? Regrettably, the answer is 
probably yes. 

We noted that there are gaps in areas of research related to early childhood education and adult 
education. It is fairly easy to understand the former, but it is harder to decipher the latter. It may 
be that the larger bodies of knowledge on secondary and postsecondary education result from the 
direct involvement of universities in these research efforts. Naturally, we would encourage a 
greater interest from school boards and districts in conducting research, and we would like to see 
emphasis placed on rigorous implementation and quality design and analysis. We see 
academic/practitioner/policy maker partnerships as a very desirable way to accommodate the 
multiple perspectives that are needed to answer these important and timely questions. 
Furthermore, the time and contribution of these various parties needs to be recognized and 
supported while there needs to be a separation of vested interests, particularly in the analysis and 
interpretation stages, to increase objectivity and to ensure credibility. 

Finally, we believe that more emphasis must be placed on implementing longitudinal research, 
whether qualitative or quantitative (preferable a mixture of the two), and that all development 
efforts be accompanied by strong evaluation components that focus on learning impact. It is a 
shame to attempt innovation and not be able to tell why it works or doesn't work. In this sense, the 
finest laboratories fore-learning research are the institutions in which it is being applied. 

Limitations and Future Directions 

The biggest unanswered question for policy makers and practitioners concerns whether e-learning 
is worth the cost. The generally positive results on achievement, motivation, and other outcomes 
suggest there is positive evidence as well as positive opinions among the stakeholders. But the 
fascinating and largely unanswered question, concerns the emphasis which is placed on 
deployment, attendant costs, and what one might take away, or not add, given the expense of an 
e-learning curriculum. While in distance learning environments this may be less of an issue since 
technology is a requirement of operation, in face-to-face and hybrid/blended learning situations 
many policy makers struggle with decisions regarding the choices between innovative pedagogy 
and innovative technology. We don't have firm answers for them yet. 



While we did not highlight them in this review, we are aware that there are promising areas of new 
development focusing on the specific applications of technology listed below. 

• Learning objects: (Downes, 2000; Friesen, 2004; McGreal & Roberts, 2001; Paquette, 2003, 2004; 
Paquette, Lundgren-Cayrol, Miara, & Guerette, 2003; Paquette & Rosea, 2002; Wiley, 2000; 2003). 

• Learning Object Repositories such as Campus Alberta Repository of Education Objects 

(CAREO: http://www. careo. ora/ j: Co-operative Learning Object Exchange (CLOE: http://cloe.on.ca/ j: 
LORNET ( http://www.lornet.ora j: and Multimedia Educational Resource for Learning and Online Teaching 
(MERLOT: ( http://www.merlot.ora/j . 

• Standardization of MetaData Projects such as CanCore ( http://www.cancore.ca/j ; The Dublin Core 
Metadata Initiative ( http://dublincore.Org/ j; World Wide Web Consortium (W3C) Metadata Activity Project 
( http://www.w3.org/Metadata/Activitv.html) ; IMS Global Learning Consortium Metadata Specification 
Project ( http://www.imsglobal.Org/metadata/ j; The Metadata Encoding and Transmission Standard Project 
(METS) ( httpV/www.loc.aov/standards/mets/ j: and the IEEE Learning Object Metadata Standard (IEEE, 
2002 ). 

• Electronic portfolios such as Learning Innovations Forum d'Innovation d'Apprentissage ( 

LIflA: http://www.lifia.ca ); Centre for the Study of Learning and Performance (CSLP) ePortfolio Project 
( http://doe.concordia.ca/cslp/ICT-ePortfolio.php j; and the ePortfolio Initiative (H. 

Barrett: http://www.electronicportfolios.Org/j . 

• Broadband Enabled Lifelong Learning Environment Project (BELLE: http://belle.netera.ca/j ; Knowledge 
Forums at the Institute for Knowledge Innovation and Technology (IKIT: http://ikit.orq/ j. Text weaving 
(Feenberg, n.d.); the learning kit for studying (Winne et al. in press); Mobile Learning (Kukulska-Hulme & 
Traxler, 2005); IMS Learning Design (Koper, 2005; Paquette, 2004); the work being done by the 
EduSource Collaboration (McGreal, et al. 2004); Fluman-Computer Interaction Research (T. 

Carey: http://watservl.uwaterloo.ca/~tcarev/hcitl lab.htmlj as well as a myriad of other promising 
developments. 

We share with others our personal enthusiasm for the potential of these inquires to bear fruit. But 
here too, if the impact of e-learning on students and educators is paramount, then we need to 
consider, as an integral part of development, whether and to what extent there are meaningful 
pedagogical advantages to the new applications. Do, in fact, students using the new technologies 
learn more, or better? Do, in fact, educators improve their teaching practices? Are the tools readily 
usable? How much professional development and training is needed? To borrow an idea from the 
film "Field of Dreams": if you build it, will they come? In other words, the promise of e-learning with 
new developments must be matched by a consideration of when, how, where, and by whom these 
new developments will be used. We suggest that needs assessments, front-end analyses, and other 
forms of usability assessment accompany tests of effectiveness, efficiency and cost-benefit. 

We emphasized Canadian primary research to determine the nature and extent of evidence in our 
country. Our ability to compare Canadian evidence with research from other countries was limited 
and indirect. We included only literature reviews of non-Canadian primary research. We agree with 
the importance of examining what is known within the nation on this important topic but we must 
also find ways to blend our knowledge with what is learned elsewhere. Some topics have a uniquely 
Canadian focus, such as language, culture, and geography, and may need to be studied separately. 
But other topics, such as the pedagogical qualities and types of technology use which impact on the 
quality and quantity of learning and motivation, not only deserve attention within Canada but also 
in combination with what we can learn from the evidence collected elsewhere. Given the magnitude 
of evidence that exists beyond our borders, and particularly from the U.S., providing a proper 
review in the time frame allotted was impossible. 

This review of e-learning attempted to go beyond traditional reviews of the empirical literature by 
including the evidence and opinions from a variety of sources. Because of the scope of this 
undertaking, its novelty, and the time constraints under which we operated, we have been able to 
provide only a rough portrait of the evidence and opinions. In doing so, we attempted to strike a 



balance between the extraordinary breadth of our review and the depth we wanted to achieve in 
conducting a detailed analysis of the documents. And even while we insured coverage of five 
distinct sources of evidence, we did not do so comprehensively for all of them. A more elaborate 
review and Argument Catalogue is called for, to explore the documents in greater depth, and to 
incorporate evidence from international efforts at primary research, in particular. 

While we incorporated a large number of variables, we are certain that a finer analysis of the 
literature would yield far more. For example, our review did not incorporate evidence on subject 
matter, duration of treatment or newer developments in technology software and courseware. In 
our recently published meta-analysis of the empirical-comparative literature of distance education 
(Bernard et al. 2004), we coded 54 study features related to the methodological quality of the 
studies, pedagogy, media and technology, context, etc. of the 236 studies we included. This meta¬ 
analysis required four years to complete at an estimated cost many times greater than the current 
review. We envision a study of the e-learning literature to require nearly that amount of time and 
resources. 

We did not examine the evidence from a theoretical perspective, in part because of the time limits, 
but primarily because there is little in the way of theory-testing research on e-learning that can be 
synthesized. The importance of doing this may depend on whether e-learning is a transparent tool 
for learning which merely delivers pedagogy and content (e.g., Clark, 1983), or a transformative 
one, which impacts how pedagogy and content are blended and delivered (e.g., Dede, 1996, 2000; 
Ullner, 1994). Investigations of computer conferencing (e.g., Bures, Abrami, & Amundsen, 2000), 
text-weaving, and knowledge forums may be worth a detailed synthesis of evidence and new 
primary evidence. Similarly, research on interactive multimedia (Mayer, 2003) and embedded 
multimedia (Chambers, Cheung, Gifford, Madden, & Slavin, in press) may explore what is unique 
about e-learning. But we also need good questions, either inductive or deductive, through which 
key propositions can be explored. 

There were also methodological challenges and shortcomings to this review and our use of an 
argument catalogue to synthesize views on e-learning. We did not weight outcomes (i.e., effect 
sizes) by their size and complexity. In traditional meta-analyses, for example, the findings from 
studies are weighted by sample size so that larger studies have more weight. We did not weight the 
primary evidence by sample size but instead treated each study equally. Similarly, we did not 
account for the size and scope of literature reviews and policy documents in aggregating evidence 
from these sources. A small review of several pages was given the same weight as a lengthy one. 
In future, we might consider given more weight to papers as a function of their scope. 

Finally, our analyses of evidence, including the primary evidence, are based merely on frequency 
analyses or vote counts of impacts without regard to the methodological quality of the evidence. 
We combined all sources of evidence, regardless of the type and quality of design, in part because 
applying more stringent criteria would have left too few documents to analyse. We also did not 
compute effect sizes and cannot estimate the magnitude of the impact of technology integration on 
achievement, motivation, dropout, and other outcomes. Vote counts tell us more about the 
consistency of effects and not about their size. But size matters for policy decisions, as do 
considerations of cost, future research should explore both. 

In most systematic reviews, especially quantitative syntheses of primary evidence, considerable 
effort is expended in judging the quality of the evidence using a plethora of methodological criteria 
often focusing on, but not necessarily limited to, a study's internal validity or the certainty with 



which causal inferences are likely. Not only did we not apply these quality judgments to the primary 
Canadian studies as exclusion criteria (given the limited number of studies which would meet 
rigorous standards), but we applied no quality criteria whatsoever in judging the other sources of 
evidence. We did, however, compare the primary evidence, literature reviews, and the opinions we 
extracted from other sources to identify similarities and differences. Nevertheless, we are sensitive 
to the possibility that a review based on an Argument Catalogue gives voice to popular perceptions 
and the Zeitgeist of current views, whether they are formed carefully or carelessly. We believe the 
advantages of a comprehensive review like ours are worth the risks and the costs of broad 
inclusivity. But the final judgment may be the readers'. 

What is Needed to Effect and Support Effective E-learning 

In education, there is the mistaken view, repeated over the generations: 1) that technology 
represents a "magical solution" to the a range of problems affecting schools and learners; and 2) 
that money for technology alone, thrown in large enough quantities at the problems of education, 
will affect the kinds of changes that are required to produce a well-informed, literate and numerate 
citizenry. It is probably true that the wide range of electronic technologies (including those that 
provide access to the Internet) that are now, and will remain available stand a better chance of 
affecting educational change than the technologies of film, television, learning machines, intelligent 
tutoring systems, etc. However, it has never been the case that money alone solves problems 
unless it is invested in equal amounts in human and physical resources. We found among the many 
coded variables that might be classified as support for e-learning, that reference to professional 
development ranked the lowest across all document types and CCL theme groups (11.16% of all 
documents examined). By contrast, nearly half of the documents we examined (47.8%) referred to 
infrastructure and logistical support or e-learning. It is arguable that the education of Canadians 
would be better served by more emphasis on preparing and training practitioners to use technology 
effectively than rushing to adopt the "technology du jour." 

Richard E. Clark (1983, 2000) has long argued that technology alone is of little importance in 
effecting achievement gains in students. Course design and instructional strategies are far more 
important aspects of effective teaching than the technology that is used. The Bernard et al. (2004) 
meta-analysis found evidence that pedagogical study features account for more variation in 
achievement outcomes that media-related study features, at least in the case of distance 
education. In addition, there are other aspects of instructional design that contribute to the 
successful implementation of any technology-based course design. There are more than 100 ISD 
models, but to illustrate the point, the steps in the model from Dick, Carey and Carey (2001) is 
shown below. The stages in their development model are: 

• Analysis 

• Design 

• Material Development 

• Implementation 

• Evaluation 

More often than not these or a similar set of steps are not considered in the design of instruction or 
the design of the larger learning systems into which technology is being integrated. The result is 
often wasted money, wasted effort and a system that does not achieve what it is intended to 
achieve. Unfortunately, the literature does not appear to document the successes or the failures of 
the use or misuse of instructional design and development models, except in the context of 
corporate training. Since the literature of corporate training was beyond the scope of this study, we 



are unable to establish this empirically. 

Evaluation of Our Methodology 

In a previous section we detailed the limitations of this review, which are mostly attributable to the 
ninety-day time frame in which it was accomplished. But what about the effectiveness of the 
approach to systematic review—the Argument Catalogue—that we have pioneered and applied 
here? Several comments about the somewhat unconventional searches that were implemented are 
appropriate. Systematically reviewing the academic literature is relatively straightforward, given 
the large range of retrieval tools that are available internationally. However, when it comes to 
media, practitioner, and policy documents, the retrieval tools are not nearly as sophisticated, 
organized or complete. Controlled vocabulary may not be used or may be quite broad due to the 
multidisciplinary nature of the index, abstracts may not be included in the index thus requiring the 
retrieval of the full document to determine its relevancy, policy and trade documents may be 
scattered across different indices, access to foreign material may be hampered, dates of coverage 
of the index may be limited and so on. This makes searching more challenging and time-consuming, 
and could perhaps, limit the reliability of the data collected. We do not believe this to have been a 
particular problem here given our focus on retrieving current Canadian material, but sufficient time 
is required to fully develop and enact proper searches in various international literatures that have 
somewhat different characteristics, search terms and the like. 

This being said, we fully believe that giving voice to the other, often not considered, sources of 
evidence, whether they are mere opinion, based on practical experience or derived from empirical 
research is important in developing a complete portrait of a field which touches Canadians at so 
many different levels, and which requires such a substantial investment in human and material 
resources. 

Elsewhere in this report we have described the primary advantages of an Argument Catalogue as: 
1) the ability to "frame" the literature for the purposes of performing other kinds of systematic 
reviews (e.g., meta-analysis); 2) showing the overlap among the various perspectives; and 3) 
revealing the gaps or discrepancies between different perspectives on a given topic or question. 
Given the limitations previously noted, but based on what we have achieved, we fully believe that 
the notion of an Argument Catalogue to be well worth pursuing, and it is our intention to continue to 
develop and perfect the methodologies associated with it. 
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